Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250626 08:22:17.449573 22785 test_util.cc:276] Using random seed: 1114856389
W20250626 08:22:18.661242 22785 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.167s user 0.410s sys 0.753s
W20250626 08:22:18.661576 22785 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.167s user 0.410s sys 0.753s
I20250626 08:22:18.663597 22785 ts_itest-base.cc:115] Starting cluster with:
I20250626 08:22:18.663780 22785 ts_itest-base.cc:116] --------------
I20250626 08:22:18.663945 22785 ts_itest-base.cc:117] 4 tablet servers
I20250626 08:22:18.664120 22785 ts_itest-base.cc:118] 3 replicas per TS
I20250626 08:22:18.664278 22785 ts_itest-base.cc:119] --------------
2025-06-26T08:22:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T08:22:18Z Disabled control of system clock
I20250626 08:22:18.704300 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:40187
--webserver_interface=127.22.64.126
--webserver_port=0
--builtin_ntp_servers=127.22.64.84:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.64.126:40187 with env {}
W20250626 08:22:19.022804 22799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:19.023371 22799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:22:19.023775 22799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:19.054854 22799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 08:22:19.055164 22799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:22:19.055366 22799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 08:22:19.055560 22799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 08:22:19.091136 22799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.64.126:40187
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:40187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.64.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:22:19.092482 22799 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:22:19.094165 22799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:19.108294 22805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:19.111384 22808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:19.108366 22806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:20.293854 22807 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:22:20.293916 22799 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:22:20.297669 22799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:22:20.300853 22799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:22:20.302325 22799 hybrid_clock.cc:648] HybridClock initialized: now 1750926140302282 us; error 63 us; skew 500 ppm
I20250626 08:22:20.303143 22799 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:20.313767 22799 webserver.cc:469] Webserver started at http://127.22.64.126:46825/ using document root <none> and password file <none>
I20250626 08:22:20.314875 22799 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:20.315117 22799 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:20.315616 22799 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:20.320324 22799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7d5e7df0b25c4922a2b9f7c0472afb40"
format_stamp: "Formatted at 2025-06-26 08:22:20 on dist-test-slave-z1j3"
I20250626 08:22:20.322018 22799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7d5e7df0b25c4922a2b9f7c0472afb40"
format_stamp: "Formatted at 2025-06-26 08:22:20 on dist-test-slave-z1j3"
I20250626 08:22:20.329644 22799 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20250626 08:22:20.336158 22815 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:20.337345 22799 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250626 08:22:20.337713 22799 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7d5e7df0b25c4922a2b9f7c0472afb40"
format_stamp: "Formatted at 2025-06-26 08:22:20 on dist-test-slave-z1j3"
I20250626 08:22:20.338093 22799 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:20.396287 22799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:20.398430 22799 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:22:20.398988 22799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:20.488967 22799 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.126:40187
I20250626 08:22:20.489053 22866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.126:40187 every 8 connection(s)
I20250626 08:22:20.491860 22799 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 08:22:20.498380 22867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:20.502446 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 22799
I20250626 08:22:20.503043 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 08:22:20.522601 22867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40: Bootstrap starting.
I20250626 08:22:20.529243 22867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:20.531225 22867 log.cc:826] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40: Log is configured to *not* fsync() on all Append() calls
I20250626 08:22:20.537297 22867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40: No bootstrap required, opened a new log
I20250626 08:22:20.559600 22867 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40187 } }
I20250626 08:22:20.560508 22867 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:20.560782 22867 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5e7df0b25c4922a2b9f7c0472afb40, State: Initialized, Role: FOLLOWER
I20250626 08:22:20.561673 22867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40187 } }
I20250626 08:22:20.562259 22867 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:22:20.562561 22867 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:22:20.563378 22867 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:20.568293 22867 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40187 } }
I20250626 08:22:20.569123 22867 leader_election.cc:304] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d5e7df0b25c4922a2b9f7c0472afb40; no voters:
I20250626 08:22:20.571359 22867 leader_election.cc:290] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:22:20.572152 22872 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:20.576285 22872 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 1 LEADER]: Becoming Leader. State: Replica: 7d5e7df0b25c4922a2b9f7c0472afb40, State: Running, Role: LEADER
I20250626 08:22:20.577059 22872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40187 } }
I20250626 08:22:20.577777 22867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:22:20.591102 22873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40187 } } }
I20250626 08:22:20.591336 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d5e7df0b25c4922a2b9f7c0472afb40. Latest consensus state: current_term: 1 leader_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40187 } } }
I20250626 08:22:20.591897 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [sys.catalog]: This master's current role is: LEADER
I20250626 08:22:20.591897 22873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [sys.catalog]: This master's current role is: LEADER
I20250626 08:22:20.599001 22881 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:22:20.621992 22881 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:22:20.641393 22881 catalog_manager.cc:1349] Generated new cluster ID: 5f6524cfabfe494ea042653af5a8f9f2
I20250626 08:22:20.641801 22881 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:22:20.660259 22881 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 08:22:20.661836 22881 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:22:20.676012 22881 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40: Generated new TSK 0
I20250626 08:22:20.677034 22881 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 08:22:20.689456 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.65:0
--local_ip_for_outbound_sockets=127.22.64.65
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:40187
--builtin_ntp_servers=127.22.64.84:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 08:22:21.048840 22891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:21.049351 22891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:22:21.049871 22891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:21.081838 22891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:22:21.082760 22891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.65
I20250626 08:22:21.118458 22891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:40187
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.65
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:22:21.119890 22891 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:22:21.121583 22891 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:21.142340 22898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:21.146971 22897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:21.146781 22900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:22.357558 22899 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:22:22.357869 22891 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:22:22.362069 22891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:22:22.365067 22891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:22:22.366520 22891 hybrid_clock.cc:648] HybridClock initialized: now 1750926142366436 us; error 88 us; skew 500 ppm
I20250626 08:22:22.367352 22891 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:22.376252 22891 webserver.cc:469] Webserver started at http://127.22.64.65:43729/ using document root <none> and password file <none>
I20250626 08:22:22.377233 22891 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:22.377440 22891 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:22.377920 22891 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:22.382683 22891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6d3a748d409e43058a9a048d809b5b19"
format_stamp: "Formatted at 2025-06-26 08:22:22 on dist-test-slave-z1j3"
I20250626 08:22:22.384076 22891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6d3a748d409e43058a9a048d809b5b19"
format_stamp: "Formatted at 2025-06-26 08:22:22 on dist-test-slave-z1j3"
I20250626 08:22:22.392114 22891 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250626 08:22:22.398684 22907 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:22.399848 22891 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250626 08:22:22.400190 22891 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6d3a748d409e43058a9a048d809b5b19"
format_stamp: "Formatted at 2025-06-26 08:22:22 on dist-test-slave-z1j3"
I20250626 08:22:22.400521 22891 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:22.453753 22891 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:22.455322 22891 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:22:22.455766 22891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:22.458850 22891 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:22.463649 22891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:22.463856 22891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:22.464135 22891 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:22.464287 22891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:22.638762 22891 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.65:42797
I20250626 08:22:22.638863 23019 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.65:42797 every 8 connection(s)
I20250626 08:22:22.641417 22891 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 08:22:22.642056 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 22891
I20250626 08:22:22.642490 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 08:22:22.650353 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.66:0
--local_ip_for_outbound_sockets=127.22.64.66
--webserver_interface=127.22.64.66
--webserver_port=0
--tserver_master_addrs=127.22.64.126:40187
--builtin_ntp_servers=127.22.64.84:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:22:22.669947 23020 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40187
I20250626 08:22:22.670401 23020 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:22.671532 23020 heartbeater.cc:507] Master 127.22.64.126:40187 requested a full tablet report, sending...
I20250626 08:22:22.674921 22832 ts_manager.cc:194] Registered new tserver with Master: 6d3a748d409e43058a9a048d809b5b19 (127.22.64.65:42797)
I20250626 08:22:22.676900 22832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.65:40313
W20250626 08:22:22.979590 23024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:22.980123 23024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:22:22.980700 23024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:23.011950 23024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:22:23.012863 23024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.66
I20250626 08:22:23.047478 23024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.64.66
--webserver_port=0
--tserver_master_addrs=127.22.64.126:40187
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.66
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:22:23.048882 23024 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:22:23.050581 23024 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:23.068022 23033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:23.680729 23020 heartbeater.cc:499] Master 127.22.64.126:40187 was elected leader, sending a full tablet report...
W20250626 08:22:23.068034 23031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:23.068101 23030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:24.247835 23032 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:22:24.247975 23024 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:22:24.251994 23024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:22:24.254333 23024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:22:24.255699 23024 hybrid_clock.cc:648] HybridClock initialized: now 1750926144255648 us; error 71 us; skew 500 ppm
I20250626 08:22:24.256584 23024 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:24.263631 23024 webserver.cc:469] Webserver started at http://127.22.64.66:33015/ using document root <none> and password file <none>
I20250626 08:22:24.264636 23024 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:24.264842 23024 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:24.265442 23024 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:24.270026 23024 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0b4716ba861f42b3843dc20dad3cd42d"
format_stamp: "Formatted at 2025-06-26 08:22:24 on dist-test-slave-z1j3"
I20250626 08:22:24.271173 23024 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0b4716ba861f42b3843dc20dad3cd42d"
format_stamp: "Formatted at 2025-06-26 08:22:24 on dist-test-slave-z1j3"
I20250626 08:22:24.278752 23024 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20250626 08:22:24.285315 23040 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:24.286540 23024 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250626 08:22:24.286850 23024 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0b4716ba861f42b3843dc20dad3cd42d"
format_stamp: "Formatted at 2025-06-26 08:22:24 on dist-test-slave-z1j3"
I20250626 08:22:24.287176 23024 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:24.340086 23024 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:24.341575 23024 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:22:24.342001 23024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:24.345067 23024 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:24.349269 23024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:24.349490 23024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:24.349797 23024 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:24.349972 23024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:24.518121 23024 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.66:33565
I20250626 08:22:24.518205 23152 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.66:33565 every 8 connection(s)
I20250626 08:22:24.520833 23024 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 08:22:24.521310 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 23024
I20250626 08:22:24.521762 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 08:22:24.528193 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.67:0
--local_ip_for_outbound_sockets=127.22.64.67
--webserver_interface=127.22.64.67
--webserver_port=0
--tserver_master_addrs=127.22.64.126:40187
--builtin_ntp_servers=127.22.64.84:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:22:24.549592 23153 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40187
I20250626 08:22:24.550202 23153 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:24.551873 23153 heartbeater.cc:507] Master 127.22.64.126:40187 requested a full tablet report, sending...
I20250626 08:22:24.554667 22832 ts_manager.cc:194] Registered new tserver with Master: 0b4716ba861f42b3843dc20dad3cd42d (127.22.64.66:33565)
I20250626 08:22:24.556133 22832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.66:43337
W20250626 08:22:24.857394 23157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:24.858055 23157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:22:24.858609 23157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:24.889948 23157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:22:24.890868 23157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.67
I20250626 08:22:24.929507 23157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.64.67
--webserver_port=0
--tserver_master_addrs=127.22.64.126:40187
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.67
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:22:24.930968 23157 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:22:24.932662 23157 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:24.950985 23164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:25.560561 23153 heartbeater.cc:499] Master 127.22.64.126:40187 was elected leader, sending a full tablet report...
W20250626 08:22:24.951015 23163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:26.199512 23166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:26.202961 23165 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1248 milliseconds
W20250626 08:22:26.203526 23157 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.256s user 0.417s sys 0.833s
W20250626 08:22:26.203922 23157 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.257s user 0.417s sys 0.833s
I20250626 08:22:26.204243 23157 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:22:26.205858 23157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:22:26.209426 23157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:22:26.210945 23157 hybrid_clock.cc:648] HybridClock initialized: now 1750926146210868 us; error 80 us; skew 500 ppm
I20250626 08:22:26.212213 23157 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:26.229523 23157 webserver.cc:469] Webserver started at http://127.22.64.67:41727/ using document root <none> and password file <none>
I20250626 08:22:26.230965 23157 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:26.231278 23157 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:26.231927 23157 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:26.239604 23157 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fb414837572c41f686b0c2c9d73870e1"
format_stamp: "Formatted at 2025-06-26 08:22:26 on dist-test-slave-z1j3"
I20250626 08:22:26.241566 23157 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fb414837572c41f686b0c2c9d73870e1"
format_stamp: "Formatted at 2025-06-26 08:22:26 on dist-test-slave-z1j3"
I20250626 08:22:26.252720 23157 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.004s sys 0.004s
I20250626 08:22:26.262140 23173 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:26.263576 23157 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.003s
I20250626 08:22:26.264046 23157 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fb414837572c41f686b0c2c9d73870e1"
format_stamp: "Formatted at 2025-06-26 08:22:26 on dist-test-slave-z1j3"
I20250626 08:22:26.264539 23157 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:26.348675 23157 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:26.350150 23157 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:22:26.350672 23157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:26.353783 23157 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:26.357978 23157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:26.358203 23157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:26.358485 23157 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:26.358678 23157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:26.535930 23157 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.67:36903
I20250626 08:22:26.536012 23285 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.67:36903 every 8 connection(s)
I20250626 08:22:26.538745 23157 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 08:22:26.548679 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 23157
I20250626 08:22:26.549217 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 08:22:26.557088 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.68:0
--local_ip_for_outbound_sockets=127.22.64.68
--webserver_interface=127.22.64.68
--webserver_port=0
--tserver_master_addrs=127.22.64.126:40187
--builtin_ntp_servers=127.22.64.84:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:22:26.566752 23286 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40187
I20250626 08:22:26.567314 23286 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:26.568751 23286 heartbeater.cc:507] Master 127.22.64.126:40187 requested a full tablet report, sending...
I20250626 08:22:26.572067 22832 ts_manager.cc:194] Registered new tserver with Master: fb414837572c41f686b0c2c9d73870e1 (127.22.64.67:36903)
I20250626 08:22:26.573407 22832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.67:37255
W20250626 08:22:26.892256 23290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:26.892789 23290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:22:26.893281 23290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:26.926047 23290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:22:26.927206 23290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.68
I20250626 08:22:26.962706 23290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.64.68
--webserver_port=0
--tserver_master_addrs=127.22.64.126:40187
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.68
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:22:26.964073 23290 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:22:26.966015 23290 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:26.986531 23296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:27.577380 23286 heartbeater.cc:499] Master 127.22.64.126:40187 was elected leader, sending a full tablet report...
W20250626 08:22:26.986968 23297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:26.988224 23299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:28.165114 23298 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:22:28.165235 23290 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:22:28.169570 23290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:22:28.172628 23290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:22:28.174096 23290 hybrid_clock.cc:648] HybridClock initialized: now 1750926148174050 us; error 59 us; skew 500 ppm
I20250626 08:22:28.175442 23290 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:28.182477 23290 webserver.cc:469] Webserver started at http://127.22.64.68:39665/ using document root <none> and password file <none>
I20250626 08:22:28.183432 23290 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:28.183730 23290 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:28.184352 23290 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:28.189157 23290 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e6c7592c1ffa49549c6e2f195a3ae867"
format_stamp: "Formatted at 2025-06-26 08:22:28 on dist-test-slave-z1j3"
I20250626 08:22:28.190310 23290 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e6c7592c1ffa49549c6e2f195a3ae867"
format_stamp: "Formatted at 2025-06-26 08:22:28 on dist-test-slave-z1j3"
I20250626 08:22:28.197688 23290 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250626 08:22:28.204164 23307 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:28.205317 23290 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20250626 08:22:28.205638 23290 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e6c7592c1ffa49549c6e2f195a3ae867"
format_stamp: "Formatted at 2025-06-26 08:22:28 on dist-test-slave-z1j3"
I20250626 08:22:28.205988 23290 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:28.284219 23290 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:28.285795 23290 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:22:28.286273 23290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:28.289412 23290 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:28.293882 23290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:28.294080 23290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:28.294322 23290 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:28.294466 23290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:28.459944 23290 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.68:39219
I20250626 08:22:28.460036 23419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.68:39219 every 8 connection(s)
I20250626 08:22:28.462545 23290 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250626 08:22:28.471139 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 23290
I20250626 08:22:28.471715 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250626 08:22:28.486563 23420 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40187
I20250626 08:22:28.487023 23420 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:28.488102 23420 heartbeater.cc:507] Master 127.22.64.126:40187 requested a full tablet report, sending...
I20250626 08:22:28.490428 22832 ts_manager.cc:194] Registered new tserver with Master: e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68:39219)
I20250626 08:22:28.491971 22832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.68:49121
I20250626 08:22:28.494462 22785 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250626 08:22:28.547847 22831 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55900:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250626 08:22:28.628008 23355 tablet_service.cc:1468] Processing CreateTablet for tablet d846b737251542d6b7ed6fbd2e85bf3d (DEFAULT_TABLE table=TestTable [id=cd169e87e6cd403db29a5e377a815cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:22:28.630312 23355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d846b737251542d6b7ed6fbd2e85bf3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:28.635605 23088 tablet_service.cc:1468] Processing CreateTablet for tablet d846b737251542d6b7ed6fbd2e85bf3d (DEFAULT_TABLE table=TestTable [id=cd169e87e6cd403db29a5e377a815cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:22:28.637786 23088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d846b737251542d6b7ed6fbd2e85bf3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:28.638741 22955 tablet_service.cc:1468] Processing CreateTablet for tablet d846b737251542d6b7ed6fbd2e85bf3d (DEFAULT_TABLE table=TestTable [id=cd169e87e6cd403db29a5e377a815cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:22:28.641010 22955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d846b737251542d6b7ed6fbd2e85bf3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:28.670066 23439 tablet_bootstrap.cc:492] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d: Bootstrap starting.
I20250626 08:22:28.671875 23440 tablet_bootstrap.cc:492] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867: Bootstrap starting.
I20250626 08:22:28.674203 23441 tablet_bootstrap.cc:492] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19: Bootstrap starting.
I20250626 08:22:28.680070 23440 tablet_bootstrap.cc:654] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:28.682245 23440 log.cc:826] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867: Log is configured to *not* fsync() on all Append() calls
I20250626 08:22:28.682544 23439 tablet_bootstrap.cc:654] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:28.683936 23441 tablet_bootstrap.cc:654] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:28.686616 23441 log.cc:826] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19: Log is configured to *not* fsync() on all Append() calls
I20250626 08:22:28.687840 23440 tablet_bootstrap.cc:492] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867: No bootstrap required, opened a new log
I20250626 08:22:28.688328 23440 ts_tablet_manager.cc:1397] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20250626 08:22:28.685228 23439 log.cc:826] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d: Log is configured to *not* fsync() on all Append() calls
I20250626 08:22:28.700017 23441 tablet_bootstrap.cc:492] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19: No bootstrap required, opened a new log
I20250626 08:22:28.700642 23441 ts_tablet_manager.cc:1397] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.007s
I20250626 08:22:28.707417 23439 tablet_bootstrap.cc:492] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d: No bootstrap required, opened a new log
I20250626 08:22:28.708432 23439 ts_tablet_manager.cc:1397] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.015s
I20250626 08:22:28.716872 23440 raft_consensus.cc:357] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
I20250626 08:22:28.718091 23440 raft_consensus.cc:383] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:28.718513 23440 raft_consensus.cc:738] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c7592c1ffa49549c6e2f195a3ae867, State: Initialized, Role: FOLLOWER
I20250626 08:22:28.719736 23440 consensus_queue.cc:260] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
I20250626 08:22:28.731731 23420 heartbeater.cc:499] Master 127.22.64.126:40187 was elected leader, sending a full tablet report...
I20250626 08:22:28.733070 23440 ts_tablet_manager.cc:1428] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867: Time spent starting tablet: real 0.045s user 0.035s sys 0.004s
I20250626 08:22:28.743757 23441 raft_consensus.cc:357] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
I20250626 08:22:28.744892 23441 raft_consensus.cc:383] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:28.745278 23441 raft_consensus.cc:738] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d3a748d409e43058a9a048d809b5b19, State: Initialized, Role: FOLLOWER
I20250626 08:22:28.746344 23441 consensus_queue.cc:260] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
I20250626 08:22:28.749619 23439 raft_consensus.cc:357] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
I20250626 08:22:28.750726 23439 raft_consensus.cc:383] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:28.751085 23439 raft_consensus.cc:738] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4716ba861f42b3843dc20dad3cd42d, State: Initialized, Role: FOLLOWER
I20250626 08:22:28.752358 23441 ts_tablet_manager.cc:1428] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19: Time spent starting tablet: real 0.051s user 0.031s sys 0.007s
I20250626 08:22:28.752118 23439 consensus_queue.cc:260] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
I20250626 08:22:28.757086 23439 ts_tablet_manager.cc:1428] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d: Time spent starting tablet: real 0.048s user 0.039s sys 0.002s
W20250626 08:22:28.781948 23154 tablet.cc:2378] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:22:28.802372 23445 raft_consensus.cc:491] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:28.803068 23445 raft_consensus.cc:513] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
I20250626 08:22:28.806499 23445 leader_election.cc:290] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b4716ba861f42b3843dc20dad3cd42d (127.22.64.66:33565), 6d3a748d409e43058a9a048d809b5b19 (127.22.64.65:42797)
I20250626 08:22:28.819593 23108 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d846b737251542d6b7ed6fbd2e85bf3d" candidate_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4716ba861f42b3843dc20dad3cd42d" is_pre_election: true
I20250626 08:22:28.819977 22975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d846b737251542d6b7ed6fbd2e85bf3d" candidate_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3a748d409e43058a9a048d809b5b19" is_pre_election: true
I20250626 08:22:28.820410 23108 raft_consensus.cc:2466] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6c7592c1ffa49549c6e2f195a3ae867 in term 0.
I20250626 08:22:28.820845 22975 raft_consensus.cc:2466] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6c7592c1ffa49549c6e2f195a3ae867 in term 0.
I20250626 08:22:28.821686 23308 leader_election.cc:304] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4716ba861f42b3843dc20dad3cd42d, e6c7592c1ffa49549c6e2f195a3ae867; no voters:
I20250626 08:22:28.822496 23445 raft_consensus.cc:2802] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:28.822799 23445 raft_consensus.cc:491] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:28.823056 23445 raft_consensus.cc:3058] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:28.827565 23445 raft_consensus.cc:513] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
I20250626 08:22:28.828972 23445 leader_election.cc:290] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [CANDIDATE]: Term 1 election: Requested vote from peers 0b4716ba861f42b3843dc20dad3cd42d (127.22.64.66:33565), 6d3a748d409e43058a9a048d809b5b19 (127.22.64.65:42797)
I20250626 08:22:28.829893 23108 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d846b737251542d6b7ed6fbd2e85bf3d" candidate_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4716ba861f42b3843dc20dad3cd42d"
I20250626 08:22:28.830046 22975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d846b737251542d6b7ed6fbd2e85bf3d" candidate_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3a748d409e43058a9a048d809b5b19"
I20250626 08:22:28.830472 23108 raft_consensus.cc:3058] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:28.830600 22975 raft_consensus.cc:3058] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:28.837836 23108 raft_consensus.cc:2466] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6c7592c1ffa49549c6e2f195a3ae867 in term 1.
I20250626 08:22:28.837836 22975 raft_consensus.cc:2466] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6c7592c1ffa49549c6e2f195a3ae867 in term 1.
I20250626 08:22:28.838976 23308 leader_election.cc:304] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4716ba861f42b3843dc20dad3cd42d, e6c7592c1ffa49549c6e2f195a3ae867; no voters:
I20250626 08:22:28.839586 23445 raft_consensus.cc:2802] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:28.841316 23445 raft_consensus.cc:695] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [term 1 LEADER]: Becoming Leader. State: Replica: e6c7592c1ffa49549c6e2f195a3ae867, State: Running, Role: LEADER
I20250626 08:22:28.842077 23445 consensus_queue.cc:237] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
I20250626 08:22:28.854084 22832 catalog_manager.cc:5582] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68). New cstate: current_term: 1 leader_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:28.891125 22785 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250626 08:22:28.895103 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6d3a748d409e43058a9a048d809b5b19 to finish bootstrapping
W20250626 08:22:28.904412 23021 tablet.cc:2378] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:22:28.912825 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0b4716ba861f42b3843dc20dad3cd42d to finish bootstrapping
I20250626 08:22:28.926126 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e6c7592c1ffa49549c6e2f195a3ae867 to finish bootstrapping
I20250626 08:22:28.937808 22785 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250626 08:22:28.941496 22785 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "d846b737251542d6b7ed6fbd2e85bf3d"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867"
rpc_addresses {
host: "127.22.64.68"
port: 39219
}
}
ts_infos {
permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d"
rpc_addresses {
host: "127.22.64.66"
port: 33565
}
}
ts_infos {
permanent_uuid: "6d3a748d409e43058a9a048d809b5b19"
rpc_addresses {
host: "127.22.64.65"
port: 42797
}
}
W20250626 08:22:28.967597 23421 tablet.cc:2378] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:22:29.291836 23445 consensus_queue.cc:1035] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:29.309055 23449 consensus_queue.cc:1035] T d846b737251542d6b7ed6fbd2e85bf3d P e6c7592c1ffa49549c6e2f195a3ae867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250626 08:22:29.314630 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 23290
W20250626 08:22:29.344175 23044 connection.cc:537] server connection from 127.22.64.68:42469 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250626 08:22:29.344491 22817 connection.cc:537] server connection from 127.22.64.68:49121 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250626 08:22:29.345575 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 22799
I20250626 08:22:29.374553 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:40187
--webserver_interface=127.22.64.126
--webserver_port=46825
--builtin_ntp_servers=127.22.64.84:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.64.126:40187 with env {}
W20250626 08:22:29.590715 23286 heartbeater.cc:646] Failed to heartbeat to 127.22.64.126:40187 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.64.126:40187: connect: Connection refused (error 111)
W20250626 08:22:29.731551 23462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:29.732197 23462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:22:29.732679 23462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:29.764474 23462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 08:22:29.764833 23462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:22:29.765095 23462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 08:22:29.765338 23462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 08:22:29.804796 23462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.64.126:40187
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:40187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.64.126
--webserver_port=46825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:22:29.806181 23462 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:22:29.807951 23462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:29.824543 23470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:30.326087 23020 heartbeater.cc:646] Failed to heartbeat to 127.22.64.126:40187 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.64.126:40187: connect: Connection refused (error 111)
W20250626 08:22:30.343472 23153 heartbeater.cc:646] Failed to heartbeat to 127.22.64.126:40187 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.64.126:40187: connect: Connection refused (error 111)
I20250626 08:22:30.806704 23478 raft_consensus.cc:491] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c7592c1ffa49549c6e2f195a3ae867)
I20250626 08:22:30.807130 23478 raft_consensus.cc:513] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
I20250626 08:22:30.809229 23478 leader_election.cc:290] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68:39219), 0b4716ba861f42b3843dc20dad3cd42d (127.22.64.66:33565)
W20250626 08:22:30.813365 22910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.64.68:39219: connect: Connection refused (error 111)
W20250626 08:22:30.817973 22910 leader_election.cc:336] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68:39219): Network error: Client connection negotiation failed: client connection to 127.22.64.68:39219: connect: Connection refused (error 111)
I20250626 08:22:30.820730 23482 raft_consensus.cc:491] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6c7592c1ffa49549c6e2f195a3ae867)
I20250626 08:22:30.821347 23482 raft_consensus.cc:513] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
I20250626 08:22:30.822847 23108 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d846b737251542d6b7ed6fbd2e85bf3d" candidate_uuid: "6d3a748d409e43058a9a048d809b5b19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0b4716ba861f42b3843dc20dad3cd42d" is_pre_election: true
I20250626 08:22:30.823309 23108 raft_consensus.cc:2466] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d3a748d409e43058a9a048d809b5b19 in term 1.
I20250626 08:22:30.824612 22908 leader_election.cc:304] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4716ba861f42b3843dc20dad3cd42d, 6d3a748d409e43058a9a048d809b5b19; no voters: e6c7592c1ffa49549c6e2f195a3ae867
I20250626 08:22:30.825266 23482 leader_election.cc:290] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68:39219), 6d3a748d409e43058a9a048d809b5b19 (127.22.64.65:42797)
I20250626 08:22:30.825830 23478 raft_consensus.cc:2802] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:22:30.826264 23478 raft_consensus.cc:491] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e6c7592c1ffa49549c6e2f195a3ae867)
I20250626 08:22:30.826753 23478 raft_consensus.cc:3058] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:22:30.829290 23043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.64.68:39219: connect: Connection refused (error 111)
I20250626 08:22:30.836074 23478 raft_consensus.cc:513] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
W20250626 08:22:30.842451 23043 leader_election.cc:336] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68:39219): Network error: Client connection negotiation failed: client connection to 127.22.64.68:39219: connect: Connection refused (error 111)
W20250626 08:22:30.843465 22910 leader_election.cc:336] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68:39219): Network error: Client connection negotiation failed: client connection to 127.22.64.68:39219: connect: Connection refused (error 111)
I20250626 08:22:30.843979 23478 leader_election.cc:290] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [CANDIDATE]: Term 2 election: Requested vote from peers e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68:39219), 0b4716ba861f42b3843dc20dad3cd42d (127.22.64.66:33565)
I20250626 08:22:30.844830 23108 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d846b737251542d6b7ed6fbd2e85bf3d" candidate_uuid: "6d3a748d409e43058a9a048d809b5b19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0b4716ba861f42b3843dc20dad3cd42d"
I20250626 08:22:30.845387 23108 raft_consensus.cc:3058] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:30.849992 22975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d846b737251542d6b7ed6fbd2e85bf3d" candidate_uuid: "0b4716ba861f42b3843dc20dad3cd42d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d3a748d409e43058a9a048d809b5b19" is_pre_election: true
I20250626 08:22:30.850658 22975 raft_consensus.cc:2391] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b4716ba861f42b3843dc20dad3cd42d in current term 2: Already voted for candidate 6d3a748d409e43058a9a048d809b5b19 in this term.
I20250626 08:22:30.851616 23043 leader_election.cc:304] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4716ba861f42b3843dc20dad3cd42d; no voters: 6d3a748d409e43058a9a048d809b5b19, e6c7592c1ffa49549c6e2f195a3ae867
I20250626 08:22:30.852178 23108 raft_consensus.cc:2466] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d3a748d409e43058a9a048d809b5b19 in term 2.
I20250626 08:22:30.852692 23482 raft_consensus.cc:2747] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:22:30.853308 22908 leader_election.cc:304] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4716ba861f42b3843dc20dad3cd42d, 6d3a748d409e43058a9a048d809b5b19; no voters: e6c7592c1ffa49549c6e2f195a3ae867
I20250626 08:22:30.853919 23478 raft_consensus.cc:2802] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:22:30.855881 23478 raft_consensus.cc:695] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 2 LEADER]: Becoming Leader. State: Replica: 6d3a748d409e43058a9a048d809b5b19, State: Running, Role: LEADER
I20250626 08:22:30.856915 23478 consensus_queue.cc:237] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
W20250626 08:22:29.824534 23472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:29.826462 23462 server_base.cc:1048] running on GCE node
W20250626 08:22:29.825250 23469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:31.079875 23462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:22:31.082626 23462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:22:31.083999 23462 hybrid_clock.cc:648] HybridClock initialized: now 1750926151083974 us; error 44 us; skew 500 ppm
I20250626 08:22:31.084794 23462 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:31.090981 23462 webserver.cc:469] Webserver started at http://127.22.64.126:46825/ using document root <none> and password file <none>
I20250626 08:22:31.091907 23462 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:31.092108 23462 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:31.100086 23462 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250626 08:22:31.104830 23493 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:31.105978 23462 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250626 08:22:31.106340 23462 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7d5e7df0b25c4922a2b9f7c0472afb40"
format_stamp: "Formatted at 2025-06-26 08:22:20 on dist-test-slave-z1j3"
I20250626 08:22:31.108384 23462 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:31.210471 23462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:31.212029 23462 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:22:31.212474 23462 kserver.cc:163] Server-wide thread pool size limit: 3276
W20250626 08:22:31.245848 22910 consensus_peers.cc:487] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 -> Peer e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68:39219): Couldn't send request to peer e6c7592c1ffa49549c6e2f195a3ae867. Status: Network error: Client connection negotiation failed: client connection to 127.22.64.68:39219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:31.278180 23108 raft_consensus.cc:1273] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 2 FOLLOWER]: Refusing update from remote peer 6d3a748d409e43058a9a048d809b5b19: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:31.280282 23478 consensus_queue.cc:1035] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:22:31.297516 23462 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.126:40187
I20250626 08:22:31.297595 23546 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.126:40187 every 8 connection(s)
I20250626 08:22:31.302155 23462 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 08:22:31.304685 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 23462
I20250626 08:22:31.305331 22785 kudu-admin-test.cc:735] Forcing unsafe config change on tserver 6d3a748d409e43058a9a048d809b5b19
I20250626 08:22:31.325778 23550 sys_catalog.cc:263] Verifying existing consensus state
I20250626 08:22:31.331369 23153 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40187
I20250626 08:22:31.331454 23020 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40187
I20250626 08:22:31.335062 23550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40: Bootstrap starting.
I20250626 08:22:31.389873 23550 log.cc:826] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40: Log is configured to *not* fsync() on all Append() calls
I20250626 08:22:31.420902 23550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:22:31.421973 23550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40: Bootstrap complete.
I20250626 08:22:31.452818 23550 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40187 } }
I20250626 08:22:31.455053 23550 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5e7df0b25c4922a2b9f7c0472afb40, State: Initialized, Role: FOLLOWER
I20250626 08:22:31.455773 23550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40187 } }
I20250626 08:22:31.456290 23550 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:22:31.456552 23550 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:22:31.456863 23550 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:31.462858 23550 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40187 } }
I20250626 08:22:31.463575 23550 leader_election.cc:304] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d5e7df0b25c4922a2b9f7c0472afb40; no voters:
I20250626 08:22:31.465764 23550 leader_election.cc:290] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 08:22:31.466115 23554 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:22:31.469545 23554 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [term 2 LEADER]: Becoming Leader. State: Replica: 7d5e7df0b25c4922a2b9f7c0472afb40, State: Running, Role: LEADER
I20250626 08:22:31.470506 23554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40187 } }
I20250626 08:22:31.471515 23550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:22:31.494313 23556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d5e7df0b25c4922a2b9f7c0472afb40. Latest consensus state: current_term: 2 leader_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40187 } } }
I20250626 08:22:31.498107 23556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [sys.catalog]: This master's current role is: LEADER
I20250626 08:22:31.494194 23555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5e7df0b25c4922a2b9f7c0472afb40" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40187 } } }
I20250626 08:22:31.498795 23555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40 [sys.catalog]: This master's current role is: LEADER
I20250626 08:22:31.509991 23564 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:22:31.528896 23564 catalog_manager.cc:671] Loaded metadata for table TestTable [id=cd169e87e6cd403db29a5e377a815cd1]
I20250626 08:22:31.551883 23564 tablet_loader.cc:96] loaded metadata for tablet d846b737251542d6b7ed6fbd2e85bf3d (table TestTable [id=cd169e87e6cd403db29a5e377a815cd1])
I20250626 08:22:31.554785 23564 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:22:31.561080 23564 catalog_manager.cc:1261] Loaded cluster ID: 5f6524cfabfe494ea042653af5a8f9f2
I20250626 08:22:31.561484 23564 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:22:31.563251 23571 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250626 08:22:31.570577 23564 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:22:31.576587 23564 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 7d5e7df0b25c4922a2b9f7c0472afb40: Loaded TSK: 0
I20250626 08:22:31.578344 23564 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 08:22:31.611382 23286 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40187
I20250626 08:22:31.617714 23509 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fb414837572c41f686b0c2c9d73870e1" instance_seqno: 1750926146490176) as {username='slave'} at 127.22.64.67:42671; Asking this server to re-register.
I20250626 08:22:31.619269 23286 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:31.619949 23286 heartbeater.cc:507] Master 127.22.64.126:40187 requested a full tablet report, sending...
I20250626 08:22:31.622781 23509 ts_manager.cc:194] Registered new tserver with Master: fb414837572c41f686b0c2c9d73870e1 (127.22.64.67:36903)
W20250626 08:22:31.721300 23552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:31.721948 23552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:31.753371 23552 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250626 08:22:32.341218 23510 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" instance_seqno: 1750926142599318) as {username='slave'} at 127.22.64.65:41247; Asking this server to re-register.
I20250626 08:22:32.342934 23509 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" instance_seqno: 1750926144476960) as {username='slave'} at 127.22.64.66:49005; Asking this server to re-register.
I20250626 08:22:32.343052 23020 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:32.343974 23020 heartbeater.cc:507] Master 127.22.64.126:40187 requested a full tablet report, sending...
I20250626 08:22:32.344537 23153 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:32.345149 23153 heartbeater.cc:507] Master 127.22.64.126:40187 requested a full tablet report, sending...
I20250626 08:22:32.348480 23509 ts_manager.cc:194] Registered new tserver with Master: 6d3a748d409e43058a9a048d809b5b19 (127.22.64.65:42797)
I20250626 08:22:32.352730 23509 catalog_manager.cc:5582] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 reported cstate change: term changed from 1 to 2, leader changed from e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68) to 6d3a748d409e43058a9a048d809b5b19 (127.22.64.65). New cstate: current_term: 2 leader_uuid: "6d3a748d409e43058a9a048d809b5b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:32.355013 23510 ts_manager.cc:194] Registered new tserver with Master: 0b4716ba861f42b3843dc20dad3cd42d (127.22.64.66:33565)
W20250626 08:22:33.139616 23552 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.338s user 0.447s sys 0.876s
W20250626 08:22:33.140007 23552 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.339s user 0.447s sys 0.876s
I20250626 08:22:33.198042 22975 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "6d3a748d409e43058a9a048d809b5b19"
tablet_id: "d846b737251542d6b7ed6fbd2e85bf3d"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d"
}
peers {
permanent_uuid: "6d3a748d409e43058a9a048d809b5b19"
}
}
from {username='slave'} at 127.0.0.1:44370
W20250626 08:22:33.199507 22975 raft_consensus.cc:2216] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } unsafe_config_change: true
I20250626 08:22:33.200670 22975 raft_consensus.cc:3053] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:22:33.200968 22975 raft_consensus.cc:738] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6d3a748d409e43058a9a048d809b5b19, State: Running, Role: LEADER
I20250626 08:22:33.201727 22975 consensus_queue.cc:260] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
I20250626 08:22:33.203145 22975 raft_consensus.cc:3058] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:34.156847 23599 raft_consensus.cc:491] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d3a748d409e43058a9a048d809b5b19)
I20250626 08:22:34.157450 23599 raft_consensus.cc:513] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } }
I20250626 08:22:34.159209 23599 leader_election.cc:290] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68:39219), 6d3a748d409e43058a9a048d809b5b19 (127.22.64.65:42797)
I20250626 08:22:34.161618 22975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d846b737251542d6b7ed6fbd2e85bf3d" candidate_uuid: "0b4716ba861f42b3843dc20dad3cd42d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d3a748d409e43058a9a048d809b5b19" is_pre_election: true
W20250626 08:22:34.164629 23043 leader_election.cc:336] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68:39219): Network error: Client connection negotiation failed: client connection to 127.22.64.68:39219: connect: Connection refused (error 111)
I20250626 08:22:34.164976 23043 leader_election.cc:304] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4716ba861f42b3843dc20dad3cd42d; no voters: 6d3a748d409e43058a9a048d809b5b19, e6c7592c1ffa49549c6e2f195a3ae867
I20250626 08:22:34.165582 23599 raft_consensus.cc:2747] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:22:34.712632 23604 raft_consensus.cc:491] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250626 08:22:34.713083 23604 raft_consensus.cc:513] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } unsafe_config_change: true
I20250626 08:22:34.714257 23604 leader_election.cc:290] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0b4716ba861f42b3843dc20dad3cd42d (127.22.64.66:33565)
I20250626 08:22:34.715590 23108 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d846b737251542d6b7ed6fbd2e85bf3d" candidate_uuid: "6d3a748d409e43058a9a048d809b5b19" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0b4716ba861f42b3843dc20dad3cd42d" is_pre_election: true
I20250626 08:22:34.716150 23108 raft_consensus.cc:2466] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d3a748d409e43058a9a048d809b5b19 in term 2.
I20250626 08:22:34.717267 22908 leader_election.cc:304] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b4716ba861f42b3843dc20dad3cd42d, 6d3a748d409e43058a9a048d809b5b19; no voters:
I20250626 08:22:34.717996 23604 raft_consensus.cc:2802] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:22:34.718297 23604 raft_consensus.cc:491] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250626 08:22:34.718583 23604 raft_consensus.cc:3058] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:22:34.723037 23604 raft_consensus.cc:513] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } unsafe_config_change: true
I20250626 08:22:34.724069 23604 leader_election.cc:290] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [CANDIDATE]: Term 4 election: Requested vote from peers 0b4716ba861f42b3843dc20dad3cd42d (127.22.64.66:33565)
I20250626 08:22:34.725090 23108 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d846b737251542d6b7ed6fbd2e85bf3d" candidate_uuid: "6d3a748d409e43058a9a048d809b5b19" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0b4716ba861f42b3843dc20dad3cd42d"
I20250626 08:22:34.725497 23108 raft_consensus.cc:3058] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 2 FOLLOWER]: Advancing to term 4
I20250626 08:22:34.730796 23108 raft_consensus.cc:2466] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d3a748d409e43058a9a048d809b5b19 in term 4.
I20250626 08:22:34.731719 22908 leader_election.cc:304] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b4716ba861f42b3843dc20dad3cd42d, 6d3a748d409e43058a9a048d809b5b19; no voters:
I20250626 08:22:34.732368 23604 raft_consensus.cc:2802] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:22:34.733207 23604 raft_consensus.cc:695] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 4 LEADER]: Becoming Leader. State: Replica: 6d3a748d409e43058a9a048d809b5b19, State: Running, Role: LEADER
I20250626 08:22:34.733816 23604 consensus_queue.cc:237] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } unsafe_config_change: true
I20250626 08:22:34.739482 23509 catalog_manager.cc:5582] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER 0b4716ba861f42b3843dc20dad3cd42d (127.22.64.66), VOTER 6d3a748d409e43058a9a048d809b5b19 (127.22.64.65). New cstate: current_term: 4 leader_uuid: "6d3a748d409e43058a9a048d809b5b19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c7592c1ffa49549c6e2f195a3ae867" member_type: VOTER last_known_addr { host: "127.22.64.68" port: 39219 } } peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } unsafe_config_change: true }
I20250626 08:22:35.308655 23108 raft_consensus.cc:1273] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 4 FOLLOWER]: Refusing update from remote peer 6d3a748d409e43058a9a048d809b5b19: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:22:35.309868 23612 consensus_queue.cc:1035] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:35.318881 23615 raft_consensus.cc:2953] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } unsafe_config_change: true }
I20250626 08:22:35.320531 23108 raft_consensus.cc:2953] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } unsafe_config_change: true }
I20250626 08:22:35.331574 23509 catalog_manager.cc:5582] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d reported cstate change: config changed from index -1 to 3, VOTER e6c7592c1ffa49549c6e2f195a3ae867 (127.22.64.68) evicted, no longer has a pending config: VOTER 0b4716ba861f42b3843dc20dad3cd42d (127.22.64.66), VOTER 6d3a748d409e43058a9a048d809b5b19 (127.22.64.65). New cstate: current_term: 4 leader_uuid: "6d3a748d409e43058a9a048d809b5b19" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } unsafe_config_change: true }
W20250626 08:22:35.341976 23509 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet d846b737251542d6b7ed6fbd2e85bf3d on TS e6c7592c1ffa49549c6e2f195a3ae867: Not found: failed to reset TS proxy: Could not find TS for UUID e6c7592c1ffa49549c6e2f195a3ae867
I20250626 08:22:35.358381 22975 consensus_queue.cc:237] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } peers { permanent_uuid: "fb414837572c41f686b0c2c9d73870e1" member_type: NON_VOTER last_known_addr { host: "127.22.64.67" port: 36903 } attrs { promote: true } } unsafe_config_change: true
I20250626 08:22:35.363302 23108 raft_consensus.cc:1273] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 4 FOLLOWER]: Refusing update from remote peer 6d3a748d409e43058a9a048d809b5b19: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250626 08:22:35.364708 23612 consensus_queue.cc:1035] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:22:35.370512 23615 raft_consensus.cc:2953] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER fb414837572c41f686b0c2c9d73870e1 (127.22.64.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } peers { permanent_uuid: "fb414837572c41f686b0c2c9d73870e1" member_type: NON_VOTER last_known_addr { host: "127.22.64.67" port: 36903 } attrs { promote: true } } unsafe_config_change: true }
I20250626 08:22:35.372169 23108 raft_consensus.cc:2953] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER fb414837572c41f686b0c2c9d73870e1 (127.22.64.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } peers { permanent_uuid: "fb414837572c41f686b0c2c9d73870e1" member_type: NON_VOTER last_known_addr { host: "127.22.64.67" port: 36903 } attrs { promote: true } } unsafe_config_change: true }
W20250626 08:22:35.375708 22909 consensus_peers.cc:487] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 -> Peer fb414837572c41f686b0c2c9d73870e1 (127.22.64.67:36903): Couldn't send request to peer fb414837572c41f686b0c2c9d73870e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d846b737251542d6b7ed6fbd2e85bf3d. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:35.379873 23496 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d846b737251542d6b7ed6fbd2e85bf3d with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 08:22:35.382910 23510 catalog_manager.cc:5582] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 reported cstate change: config changed from index 3 to 5, NON_VOTER fb414837572c41f686b0c2c9d73870e1 (127.22.64.67) added. New cstate: current_term: 4 leader_uuid: "6d3a748d409e43058a9a048d809b5b19" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb414837572c41f686b0c2c9d73870e1" member_type: NON_VOTER last_known_addr { host: "127.22.64.67" port: 36903 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250626 08:22:35.394402 23495 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet d846b737251542d6b7ed6fbd2e85bf3d on TS e6c7592c1ffa49549c6e2f195a3ae867 failed: Not found: failed to reset TS proxy: Could not find TS for UUID e6c7592c1ffa49549c6e2f195a3ae867
I20250626 08:22:35.943051 23626 ts_tablet_manager.cc:927] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1: Initiating tablet copy from peer 6d3a748d409e43058a9a048d809b5b19 (127.22.64.65:42797)
I20250626 08:22:35.945667 23626 tablet_copy_client.cc:323] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.64.65:42797
I20250626 08:22:35.956231 22995 tablet_copy_service.cc:140] P 6d3a748d409e43058a9a048d809b5b19: Received BeginTabletCopySession request for tablet d846b737251542d6b7ed6fbd2e85bf3d from peer fb414837572c41f686b0c2c9d73870e1 ({username='slave'} at 127.22.64.67:58459)
I20250626 08:22:35.956694 22995 tablet_copy_service.cc:161] P 6d3a748d409e43058a9a048d809b5b19: Beginning new tablet copy session on tablet d846b737251542d6b7ed6fbd2e85bf3d from peer fb414837572c41f686b0c2c9d73870e1 at {username='slave'} at 127.22.64.67:58459: session id = fb414837572c41f686b0c2c9d73870e1-d846b737251542d6b7ed6fbd2e85bf3d
I20250626 08:22:35.961647 22995 tablet_copy_source_session.cc:215] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:22:35.967419 23626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d846b737251542d6b7ed6fbd2e85bf3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:35.986500 23626 tablet_copy_client.cc:806] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1: tablet copy: Starting download of 0 data blocks...
I20250626 08:22:35.987154 23626 tablet_copy_client.cc:670] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1: tablet copy: Starting download of 1 WAL segments...
I20250626 08:22:35.991295 23626 tablet_copy_client.cc:538] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:22:35.997288 23626 tablet_bootstrap.cc:492] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1: Bootstrap starting.
I20250626 08:22:36.009630 23626 log.cc:826] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1: Log is configured to *not* fsync() on all Append() calls
I20250626 08:22:36.020928 23626 tablet_bootstrap.cc:492] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:22:36.021745 23626 tablet_bootstrap.cc:492] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1: Bootstrap complete.
I20250626 08:22:36.022387 23626 ts_tablet_manager.cc:1397] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.005s
I20250626 08:22:36.040272 23626 raft_consensus.cc:357] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } peers { permanent_uuid: "fb414837572c41f686b0c2c9d73870e1" member_type: NON_VOTER last_known_addr { host: "127.22.64.67" port: 36903 } attrs { promote: true } } unsafe_config_change: true
I20250626 08:22:36.041177 23626 raft_consensus.cc:738] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: fb414837572c41f686b0c2c9d73870e1, State: Initialized, Role: LEARNER
I20250626 08:22:36.041795 23626 consensus_queue.cc:260] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } peers { permanent_uuid: "fb414837572c41f686b0c2c9d73870e1" member_type: NON_VOTER last_known_addr { host: "127.22.64.67" port: 36903 } attrs { promote: true } } unsafe_config_change: true
I20250626 08:22:36.045408 23626 ts_tablet_manager.cc:1428] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1: Time spent starting tablet: real 0.023s user 0.021s sys 0.000s
I20250626 08:22:36.047029 22995 tablet_copy_service.cc:342] P 6d3a748d409e43058a9a048d809b5b19: Request end of tablet copy session fb414837572c41f686b0c2c9d73870e1-d846b737251542d6b7ed6fbd2e85bf3d received from {username='slave'} at 127.22.64.67:58459
I20250626 08:22:36.047390 22995 tablet_copy_service.cc:434] P 6d3a748d409e43058a9a048d809b5b19: ending tablet copy session fb414837572c41f686b0c2c9d73870e1-d846b737251542d6b7ed6fbd2e85bf3d on tablet d846b737251542d6b7ed6fbd2e85bf3d with peer fb414837572c41f686b0c2c9d73870e1
I20250626 08:22:36.441035 23241 raft_consensus.cc:1215] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250626 08:22:36.560905 23495 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet d846b737251542d6b7ed6fbd2e85bf3d on TS e6c7592c1ffa49549c6e2f195a3ae867 failed: Not found: failed to reset TS proxy: Could not find TS for UUID e6c7592c1ffa49549c6e2f195a3ae867
I20250626 08:22:37.000802 23633 raft_consensus.cc:1062] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19: attempting to promote NON_VOTER fb414837572c41f686b0c2c9d73870e1 to VOTER
I20250626 08:22:37.002441 23633 consensus_queue.cc:237] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } peers { permanent_uuid: "fb414837572c41f686b0c2c9d73870e1" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 36903 } attrs { promote: false } } unsafe_config_change: true
I20250626 08:22:37.007467 23241 raft_consensus.cc:1273] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1 [term 4 LEARNER]: Refusing update from remote peer 6d3a748d409e43058a9a048d809b5b19: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:22:37.007642 23108 raft_consensus.cc:1273] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 4 FOLLOWER]: Refusing update from remote peer 6d3a748d409e43058a9a048d809b5b19: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:22:37.008638 23633 consensus_queue.cc:1035] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb414837572c41f686b0c2c9d73870e1" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 36903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:22:37.009436 23634 consensus_queue.cc:1035] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:22:37.015664 23633 raft_consensus.cc:2953] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, fb414837572c41f686b0c2c9d73870e1 (127.22.64.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } peers { permanent_uuid: "fb414837572c41f686b0c2c9d73870e1" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 36903 } attrs { promote: false } } unsafe_config_change: true }
I20250626 08:22:37.017438 23108 raft_consensus.cc:2953] T d846b737251542d6b7ed6fbd2e85bf3d P 0b4716ba861f42b3843dc20dad3cd42d [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, fb414837572c41f686b0c2c9d73870e1 (127.22.64.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } peers { permanent_uuid: "fb414837572c41f686b0c2c9d73870e1" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 36903 } attrs { promote: false } } unsafe_config_change: true }
I20250626 08:22:37.019119 23240 raft_consensus.cc:2953] T d846b737251542d6b7ed6fbd2e85bf3d P fb414837572c41f686b0c2c9d73870e1 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, fb414837572c41f686b0c2c9d73870e1 (127.22.64.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } } peers { permanent_uuid: "fb414837572c41f686b0c2c9d73870e1" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 36903 } attrs { promote: false } } unsafe_config_change: true }
I20250626 08:22:37.027969 23510 catalog_manager.cc:5582] T d846b737251542d6b7ed6fbd2e85bf3d P 6d3a748d409e43058a9a048d809b5b19 reported cstate change: config changed from index 5 to 6, fb414837572c41f686b0c2c9d73870e1 (127.22.64.67) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "6d3a748d409e43058a9a048d809b5b19" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 33565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d3a748d409e43058a9a048d809b5b19" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 42797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb414837572c41f686b0c2c9d73870e1" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 36903 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250626 08:22:37.143224 22785 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250626 08:22:37.161751 22785 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "d846b737251542d6b7ed6fbd2e85bf3d"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "0b4716ba861f42b3843dc20dad3cd42d"
rpc_addresses {
host: "127.22.64.66"
port: 33565
}
}
ts_infos {
permanent_uuid: "6d3a748d409e43058a9a048d809b5b19"
rpc_addresses {
host: "127.22.64.65"
port: 42797
}
}
ts_infos {
permanent_uuid: "fb414837572c41f686b0c2c9d73870e1"
rpc_addresses {
host: "127.22.64.67"
port: 36903
}
}
I20250626 08:22:37.165831 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 22891
I20250626 08:22:37.193473 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 23024
I20250626 08:22:37.222973 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 23157
I20250626 08:22:37.249320 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 23462
2025-06-26T08:22:37Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19863 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250626 08:22:37.310894 22785 test_util.cc:276] Using random seed: 1134717842
I20250626 08:22:37.316920 22785 ts_itest-base.cc:115] Starting cluster with:
I20250626 08:22:37.317097 22785 ts_itest-base.cc:116] --------------
I20250626 08:22:37.317269 22785 ts_itest-base.cc:117] 3 tablet servers
I20250626 08:22:37.317414 22785 ts_itest-base.cc:118] 3 replicas per TS
I20250626 08:22:37.317559 22785 ts_itest-base.cc:119] --------------
2025-06-26T08:22:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T08:22:37Z Disabled control of system clock
I20250626 08:22:37.357378 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:37225
--webserver_interface=127.22.64.126
--webserver_port=0
--builtin_ntp_servers=127.22.64.84:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.64.126:37225
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250626 08:22:37.661866 23654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:37.662519 23654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:22:37.663035 23654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:37.695107 23654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 08:22:37.695461 23654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:22:37.695720 23654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 08:22:37.695971 23654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 08:22:37.731900 23654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.22.64.126:37225
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:37225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.64.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:22:37.733310 23654 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:22:37.735065 23654 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:37.749498 23662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:37.749485 23661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:38.950539 23663 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250626 08:22:38.952618 23664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:38.952711 23654 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:22:38.956017 23654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:22:38.959492 23654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:22:38.961174 23654 hybrid_clock.cc:648] HybridClock initialized: now 1750926158961151 us; error 41 us; skew 500 ppm
I20250626 08:22:38.962054 23654 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:38.969077 23654 webserver.cc:469] Webserver started at http://127.22.64.126:41841/ using document root <none> and password file <none>
I20250626 08:22:38.970276 23654 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:38.970497 23654 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:38.970952 23654 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:38.975872 23654 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b631118ebd7148ad8624a2459fdb28b9"
format_stamp: "Formatted at 2025-06-26 08:22:38 on dist-test-slave-z1j3"
I20250626 08:22:38.977085 23654 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b631118ebd7148ad8624a2459fdb28b9"
format_stamp: "Formatted at 2025-06-26 08:22:38 on dist-test-slave-z1j3"
I20250626 08:22:38.984892 23654 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250626 08:22:38.991190 23671 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:38.992401 23654 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250626 08:22:38.992767 23654 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b631118ebd7148ad8624a2459fdb28b9"
format_stamp: "Formatted at 2025-06-26 08:22:38 on dist-test-slave-z1j3"
I20250626 08:22:38.993125 23654 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:39.045074 23654 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:39.046638 23654 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:22:39.047112 23654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:39.124676 23654 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.126:37225
I20250626 08:22:39.124742 23722 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.126:37225 every 8 connection(s)
I20250626 08:22:39.127749 23654 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 08:22:39.133153 23723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:39.136369 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 23654
I20250626 08:22:39.137095 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 08:22:39.156065 23723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9: Bootstrap starting.
I20250626 08:22:39.162714 23723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.165557 23723 log.cc:826] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9: Log is configured to *not* fsync() on all Append() calls
I20250626 08:22:39.171036 23723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9: No bootstrap required, opened a new log
I20250626 08:22:39.190362 23723 raft_consensus.cc:357] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b631118ebd7148ad8624a2459fdb28b9" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 37225 } }
I20250626 08:22:39.191092 23723 raft_consensus.cc:383] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.191295 23723 raft_consensus.cc:738] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b631118ebd7148ad8624a2459fdb28b9, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.192052 23723 consensus_queue.cc:260] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b631118ebd7148ad8624a2459fdb28b9" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 37225 } }
I20250626 08:22:39.192557 23723 raft_consensus.cc:397] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:22:39.192806 23723 raft_consensus.cc:491] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:22:39.193073 23723 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.279562 23723 raft_consensus.cc:513] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b631118ebd7148ad8624a2459fdb28b9" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 37225 } }
I20250626 08:22:39.280434 23723 leader_election.cc:304] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b631118ebd7148ad8624a2459fdb28b9; no voters:
I20250626 08:22:39.282434 23723 leader_election.cc:290] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:22:39.283257 23728 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:39.286501 23728 raft_consensus.cc:695] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [term 1 LEADER]: Becoming Leader. State: Replica: b631118ebd7148ad8624a2459fdb28b9, State: Running, Role: LEADER
I20250626 08:22:39.287360 23728 consensus_queue.cc:237] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b631118ebd7148ad8624a2459fdb28b9" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 37225 } }
I20250626 08:22:39.288431 23723 sys_catalog.cc:564] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:22:39.299678 23730 sys_catalog.cc:455] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b631118ebd7148ad8624a2459fdb28b9. Latest consensus state: current_term: 1 leader_uuid: "b631118ebd7148ad8624a2459fdb28b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b631118ebd7148ad8624a2459fdb28b9" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 37225 } } }
I20250626 08:22:39.300740 23730 sys_catalog.cc:458] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [sys.catalog]: This master's current role is: LEADER
I20250626 08:22:39.302747 23729 sys_catalog.cc:455] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b631118ebd7148ad8624a2459fdb28b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b631118ebd7148ad8624a2459fdb28b9" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 37225 } } }
I20250626 08:22:39.303670 23729 sys_catalog.cc:458] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9 [sys.catalog]: This master's current role is: LEADER
W20250626 08:22:39.304895 23677 tablet.cc:2378] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:22:39.312110 23736 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:22:39.323621 23736 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:22:39.340817 23736 catalog_manager.cc:1349] Generated new cluster ID: 05e0df3b73544ed4ba2f049fe26e7495
I20250626 08:22:39.341166 23736 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:22:39.363582 23736 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 08:22:39.365152 23736 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:22:39.378957 23736 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b631118ebd7148ad8624a2459fdb28b9: Generated new TSK 0
I20250626 08:22:39.380015 23736 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 08:22:39.399791 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.65:0
--local_ip_for_outbound_sockets=127.22.64.65
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:37225
--builtin_ntp_servers=127.22.64.84:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250626 08:22:39.726032 23747 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250626 08:22:39.726831 23747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:39.727121 23747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:22:39.727648 23747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:39.761207 23747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:22:39.762120 23747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.65
I20250626 08:22:39.803433 23747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:37225
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.65
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:22:39.805012 23747 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:22:39.806860 23747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:39.826093 23754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:39.826195 23753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:41.223590 23752 debug-util.cc:398] Leaking SignalData structure 0x7b0800015620 after lost signal to thread 23747
W20250626 08:22:41.475337 23747 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.654s user 0.518s sys 1.129s
W20250626 08:22:41.476922 23747 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.656s user 0.518s sys 1.130s
W20250626 08:22:41.477770 23756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:41.480877 23747 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250626 08:22:41.480868 23755 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1657 milliseconds
I20250626 08:22:41.482246 23747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:22:41.485031 23747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:22:41.486503 23747 hybrid_clock.cc:648] HybridClock initialized: now 1750926161486426 us; error 82 us; skew 500 ppm
I20250626 08:22:41.487702 23747 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:41.495740 23747 webserver.cc:469] Webserver started at http://127.22.64.65:42053/ using document root <none> and password file <none>
I20250626 08:22:41.497046 23747 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:41.497327 23747 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:41.497941 23747 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:41.504684 23747 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "796526474f254c0db0c39a41c783b7de"
format_stamp: "Formatted at 2025-06-26 08:22:41 on dist-test-slave-z1j3"
I20250626 08:22:41.506285 23747 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "796526474f254c0db0c39a41c783b7de"
format_stamp: "Formatted at 2025-06-26 08:22:41 on dist-test-slave-z1j3"
I20250626 08:22:41.516403 23747 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.002s sys 0.006s
I20250626 08:22:41.524307 23763 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:41.525592 23747 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250626 08:22:41.526002 23747 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "796526474f254c0db0c39a41c783b7de"
format_stamp: "Formatted at 2025-06-26 08:22:41 on dist-test-slave-z1j3"
I20250626 08:22:41.526502 23747 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:41.594616 23747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:41.596645 23747 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:22:41.597214 23747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:41.600746 23747 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:41.606504 23747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:41.606818 23747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:41.607125 23747 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:41.607339 23747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:41.763257 23747 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.65:37571
I20250626 08:22:41.763310 23875 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.65:37571 every 8 connection(s)
I20250626 08:22:41.767014 23747 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 08:22:41.775002 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 23747
I20250626 08:22:41.775453 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 08:22:41.781075 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.66:0
--local_ip_for_outbound_sockets=127.22.64.66
--webserver_interface=127.22.64.66
--webserver_port=0
--tserver_master_addrs=127.22.64.126:37225
--builtin_ntp_servers=127.22.64.84:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250626 08:22:41.798326 23876 heartbeater.cc:344] Connected to a master server at 127.22.64.126:37225
I20250626 08:22:41.798975 23876 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:41.800486 23876 heartbeater.cc:507] Master 127.22.64.126:37225 requested a full tablet report, sending...
I20250626 08:22:41.804056 23688 ts_manager.cc:194] Registered new tserver with Master: 796526474f254c0db0c39a41c783b7de (127.22.64.65:37571)
I20250626 08:22:41.807173 23688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.65:44687
W20250626 08:22:42.107074 23880 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250626 08:22:42.107726 23880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:42.107985 23880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:22:42.108466 23880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:42.140602 23880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:22:42.141479 23880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.66
I20250626 08:22:42.177147 23880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.64.66
--webserver_port=0
--tserver_master_addrs=127.22.64.126:37225
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.66
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:22:42.178627 23880 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:22:42.180296 23880 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:42.198179 23886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:42.812537 23876 heartbeater.cc:499] Master 127.22.64.126:37225 was elected leader, sending a full tablet report...
W20250626 08:22:42.198275 23887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:42.199590 23889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:43.372532 23888 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:22:43.372617 23880 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:22:43.377158 23880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:22:43.380007 23880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:22:43.381455 23880 hybrid_clock.cc:648] HybridClock initialized: now 1750926163381419 us; error 57 us; skew 500 ppm
I20250626 08:22:43.382356 23880 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:43.395257 23880 webserver.cc:469] Webserver started at http://127.22.64.66:39133/ using document root <none> and password file <none>
I20250626 08:22:43.396260 23880 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:43.396471 23880 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:43.396939 23880 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:43.401990 23880 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "48eaaf272ca8455c8e69c89b34e0ccb4"
format_stamp: "Formatted at 2025-06-26 08:22:43 on dist-test-slave-z1j3"
I20250626 08:22:43.403262 23880 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "48eaaf272ca8455c8e69c89b34e0ccb4"
format_stamp: "Formatted at 2025-06-26 08:22:43 on dist-test-slave-z1j3"
I20250626 08:22:43.410893 23880 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250626 08:22:43.417490 23896 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:43.418771 23880 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250626 08:22:43.419114 23880 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "48eaaf272ca8455c8e69c89b34e0ccb4"
format_stamp: "Formatted at 2025-06-26 08:22:43 on dist-test-slave-z1j3"
I20250626 08:22:43.419463 23880 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:43.480221 23880 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:43.481760 23880 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:22:43.482249 23880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:43.484968 23880 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:43.489271 23880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:43.489521 23880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:43.489781 23880 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:43.489969 23880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:43.628621 23880 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.66:42705
I20250626 08:22:43.628719 24008 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.66:42705 every 8 connection(s)
I20250626 08:22:43.631251 23880 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 08:22:43.634160 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 23880
I20250626 08:22:43.634794 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 08:22:43.641948 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.67:0
--local_ip_for_outbound_sockets=127.22.64.67
--webserver_interface=127.22.64.67
--webserver_port=0
--tserver_master_addrs=127.22.64.126:37225
--builtin_ntp_servers=127.22.64.84:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250626 08:22:43.653744 24009 heartbeater.cc:344] Connected to a master server at 127.22.64.126:37225
I20250626 08:22:43.654268 24009 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:43.655380 24009 heartbeater.cc:507] Master 127.22.64.126:37225 requested a full tablet report, sending...
I20250626 08:22:43.657776 23688 ts_manager.cc:194] Registered new tserver with Master: 48eaaf272ca8455c8e69c89b34e0ccb4 (127.22.64.66:42705)
I20250626 08:22:43.659152 23688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.66:40555
W20250626 08:22:43.954265 24013 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250626 08:22:43.954991 24013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:43.955214 24013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:22:43.955662 24013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:43.988868 24013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:22:43.989703 24013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.67
I20250626 08:22:44.025959 24013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.64.67
--webserver_port=0
--tserver_master_addrs=127.22.64.126:37225
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.67
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:22:44.027421 24013 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:22:44.029143 24013 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:44.045032 24019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:44.662732 24009 heartbeater.cc:499] Master 127.22.64.126:37225 was elected leader, sending a full tablet report...
W20250626 08:22:44.045392 24020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:44.047747 24013 server_base.cc:1048] running on GCE node
W20250626 08:22:44.045851 24022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:45.215619 24013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:22:45.217962 24013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:22:45.219384 24013 hybrid_clock.cc:648] HybridClock initialized: now 1750926165219353 us; error 78 us; skew 500 ppm
I20250626 08:22:45.220314 24013 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:45.226877 24013 webserver.cc:469] Webserver started at http://127.22.64.67:38271/ using document root <none> and password file <none>
I20250626 08:22:45.227891 24013 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:45.228121 24013 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:45.228642 24013 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:45.233289 24013 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "778031abc17a4e9393bcb0dcc2c17c73"
format_stamp: "Formatted at 2025-06-26 08:22:45 on dist-test-slave-z1j3"
I20250626 08:22:45.234551 24013 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "778031abc17a4e9393bcb0dcc2c17c73"
format_stamp: "Formatted at 2025-06-26 08:22:45 on dist-test-slave-z1j3"
I20250626 08:22:45.242197 24013 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250626 08:22:45.248097 24029 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:45.249236 24013 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250626 08:22:45.249590 24013 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "778031abc17a4e9393bcb0dcc2c17c73"
format_stamp: "Formatted at 2025-06-26 08:22:45 on dist-test-slave-z1j3"
I20250626 08:22:45.249970 24013 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:45.298692 24013 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:45.300190 24013 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:22:45.300678 24013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:45.303241 24013 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:45.307725 24013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:45.307972 24013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:45.308212 24013 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:45.308383 24013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:45.446038 24013 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.67:40877
I20250626 08:22:45.446121 24141 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.67:40877 every 8 connection(s)
I20250626 08:22:45.448689 24013 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 08:22:45.450577 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 24013
I20250626 08:22:45.451155 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 08:22:45.471014 24142 heartbeater.cc:344] Connected to a master server at 127.22.64.126:37225
I20250626 08:22:45.471477 24142 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:45.472674 24142 heartbeater.cc:507] Master 127.22.64.126:37225 requested a full tablet report, sending...
I20250626 08:22:45.474835 23688 ts_manager.cc:194] Registered new tserver with Master: 778031abc17a4e9393bcb0dcc2c17c73 (127.22.64.67:40877)
I20250626 08:22:45.476125 23688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.67:42999
I20250626 08:22:45.487056 22785 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 08:22:45.524588 23688 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49150:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250626 08:22:45.544193 23688 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 08:22:45.602252 23944 tablet_service.cc:1468] Processing CreateTablet for tablet 2efe182669f347edb91170c84432fe4a (DEFAULT_TABLE table=TestTable [id=49e7515b7ac64259b8fba84265155fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:22:45.604112 23944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efe182669f347edb91170c84432fe4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:45.606727 23811 tablet_service.cc:1468] Processing CreateTablet for tablet 2efe182669f347edb91170c84432fe4a (DEFAULT_TABLE table=TestTable [id=49e7515b7ac64259b8fba84265155fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:22:45.608321 23811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efe182669f347edb91170c84432fe4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:45.615355 24077 tablet_service.cc:1468] Processing CreateTablet for tablet 2efe182669f347edb91170c84432fe4a (DEFAULT_TABLE table=TestTable [id=49e7515b7ac64259b8fba84265155fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:22:45.616912 24077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efe182669f347edb91170c84432fe4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:45.652184 24163 tablet_bootstrap.cc:492] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73: Bootstrap starting.
I20250626 08:22:45.653983 24161 tablet_bootstrap.cc:492] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4: Bootstrap starting.
I20250626 08:22:45.659806 24162 tablet_bootstrap.cc:492] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de: Bootstrap starting.
I20250626 08:22:45.662371 24163 tablet_bootstrap.cc:654] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.663829 24161 tablet_bootstrap.cc:654] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.665020 24163 log.cc:826] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73: Log is configured to *not* fsync() on all Append() calls
I20250626 08:22:45.670197 24162 tablet_bootstrap.cc:654] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.673101 24161 log.cc:826] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4: Log is configured to *not* fsync() on all Append() calls
I20250626 08:22:45.673673 24162 log.cc:826] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de: Log is configured to *not* fsync() on all Append() calls
I20250626 08:22:45.682057 24161 tablet_bootstrap.cc:492] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4: No bootstrap required, opened a new log
I20250626 08:22:45.682596 24161 ts_tablet_manager.cc:1397] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.005s
I20250626 08:22:45.683696 24162 tablet_bootstrap.cc:492] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de: No bootstrap required, opened a new log
I20250626 08:22:45.684383 24162 ts_tablet_manager.cc:1397] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de: Time spent bootstrapping tablet: real 0.025s user 0.017s sys 0.004s
I20250626 08:22:45.684859 24163 tablet_bootstrap.cc:492] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73: No bootstrap required, opened a new log
I20250626 08:22:45.685392 24163 ts_tablet_manager.cc:1397] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.014s
I20250626 08:22:45.705952 24161 raft_consensus.cc:357] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796526474f254c0db0c39a41c783b7de" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37571 } } peers { permanent_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42705 } } peers { permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 } }
I20250626 08:22:45.706764 24161 raft_consensus.cc:738] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48eaaf272ca8455c8e69c89b34e0ccb4, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.707410 24161 consensus_queue.cc:260] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796526474f254c0db0c39a41c783b7de" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37571 } } peers { permanent_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42705 } } peers { permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 } }
I20250626 08:22:45.712098 24161 ts_tablet_manager.cc:1428] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4: Time spent starting tablet: real 0.029s user 0.026s sys 0.000s
I20250626 08:22:45.712551 24163 raft_consensus.cc:357] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796526474f254c0db0c39a41c783b7de" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37571 } } peers { permanent_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42705 } } peers { permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 } }
I20250626 08:22:45.713483 24163 raft_consensus.cc:738] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778031abc17a4e9393bcb0dcc2c17c73, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.713825 24162 raft_consensus.cc:357] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796526474f254c0db0c39a41c783b7de" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37571 } } peers { permanent_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42705 } } peers { permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 } }
I20250626 08:22:45.714293 24163 consensus_queue.cc:260] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796526474f254c0db0c39a41c783b7de" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37571 } } peers { permanent_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42705 } } peers { permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 } }
I20250626 08:22:45.714776 24162 raft_consensus.cc:738] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796526474f254c0db0c39a41c783b7de, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.717569 24142 heartbeater.cc:499] Master 127.22.64.126:37225 was elected leader, sending a full tablet report...
I20250626 08:22:45.715471 24162 consensus_queue.cc:260] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796526474f254c0db0c39a41c783b7de" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37571 } } peers { permanent_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42705 } } peers { permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 } }
I20250626 08:22:45.719118 24163 ts_tablet_manager.cc:1428] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73: Time spent starting tablet: real 0.033s user 0.027s sys 0.008s
I20250626 08:22:45.722290 24162 ts_tablet_manager.cc:1428] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de: Time spent starting tablet: real 0.038s user 0.030s sys 0.004s
I20250626 08:22:45.759038 22785 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 08:22:45.762765 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 796526474f254c0db0c39a41c783b7de to finish bootstrapping
W20250626 08:22:45.775979 23877 tablet.cc:2378] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:22:45.779335 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 48eaaf272ca8455c8e69c89b34e0ccb4 to finish bootstrapping
I20250626 08:22:45.794124 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 778031abc17a4e9393bcb0dcc2c17c73 to finish bootstrapping
I20250626 08:22:45.840145 23831 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2efe182669f347edb91170c84432fe4a"
dest_uuid: "796526474f254c0db0c39a41c783b7de"
from {username='slave'} at 127.0.0.1:58292
I20250626 08:22:45.840744 23831 raft_consensus.cc:491] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:45.841081 23831 raft_consensus.cc:3058] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.845574 23831 raft_consensus.cc:513] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796526474f254c0db0c39a41c783b7de" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37571 } } peers { permanent_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42705 } } peers { permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 } }
I20250626 08:22:45.848098 23831 leader_election.cc:290] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [CANDIDATE]: Term 1 election: Requested vote from peers 48eaaf272ca8455c8e69c89b34e0ccb4 (127.22.64.66:42705), 778031abc17a4e9393bcb0dcc2c17c73 (127.22.64.67:40877)
I20250626 08:22:45.857914 22785 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250626 08:22:45.861860 23964 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2efe182669f347edb91170c84432fe4a" candidate_uuid: "796526474f254c0db0c39a41c783b7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4"
I20250626 08:22:45.862017 24097 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2efe182669f347edb91170c84432fe4a" candidate_uuid: "796526474f254c0db0c39a41c783b7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "778031abc17a4e9393bcb0dcc2c17c73"
I20250626 08:22:45.862587 23964 raft_consensus.cc:3058] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.862660 24097 raft_consensus.cc:3058] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.867182 23964 raft_consensus.cc:2466] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 796526474f254c0db0c39a41c783b7de in term 1.
I20250626 08:22:45.867182 24097 raft_consensus.cc:2466] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 796526474f254c0db0c39a41c783b7de in term 1.
I20250626 08:22:45.868389 23767 leader_election.cc:304] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48eaaf272ca8455c8e69c89b34e0ccb4, 796526474f254c0db0c39a41c783b7de; no voters:
I20250626 08:22:45.869236 24169 raft_consensus.cc:2802] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:45.870937 24169 raft_consensus.cc:695] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 1 LEADER]: Becoming Leader. State: Replica: 796526474f254c0db0c39a41c783b7de, State: Running, Role: LEADER
I20250626 08:22:45.871861 24169 consensus_queue.cc:237] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796526474f254c0db0c39a41c783b7de" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37571 } } peers { permanent_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42705 } } peers { permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 } }
I20250626 08:22:45.882848 23687 catalog_manager.cc:5582] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 796526474f254c0db0c39a41c783b7de (127.22.64.65). New cstate: current_term: 1 leader_uuid: "796526474f254c0db0c39a41c783b7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796526474f254c0db0c39a41c783b7de" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 } health_report { overall_health: UNKNOWN } } }
W20250626 08:22:45.888471 24010 tablet.cc:2378] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 08:22:45.954072 24143 tablet.cc:2378] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:22:45.964154 22785 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250626 08:22:46.170295 22785 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250626 08:22:46.300796 24169 consensus_queue.cc:1035] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:46.316494 24169 consensus_queue.cc:1035] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:48.222940 23831 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2efe182669f347edb91170c84432fe4a"
dest_uuid: "796526474f254c0db0c39a41c783b7de"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58300
I20250626 08:22:48.223475 23831 raft_consensus.cc:604] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 1 LEADER]: Received request to transfer leadership
I20250626 08:22:48.489952 24188 raft_consensus.cc:991] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de: : Instructing follower 48eaaf272ca8455c8e69c89b34e0ccb4 to start an election
I20250626 08:22:48.490424 24206 raft_consensus.cc:1079] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 1 LEADER]: Signalling peer 48eaaf272ca8455c8e69c89b34e0ccb4 to start an election
I20250626 08:22:48.492105 23964 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2efe182669f347edb91170c84432fe4a"
dest_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4"
from {username='slave'} at 127.22.64.65:52431
I20250626 08:22:48.492659 23964 raft_consensus.cc:491] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:48.492947 23964 raft_consensus.cc:3058] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:48.497139 23964 raft_consensus.cc:513] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796526474f254c0db0c39a41c783b7de" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37571 } } peers { permanent_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42705 } } peers { permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 } }
I20250626 08:22:48.499337 23964 leader_election.cc:290] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [CANDIDATE]: Term 2 election: Requested vote from peers 796526474f254c0db0c39a41c783b7de (127.22.64.65:37571), 778031abc17a4e9393bcb0dcc2c17c73 (127.22.64.67:40877)
I20250626 08:22:48.510269 23831 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2efe182669f347edb91170c84432fe4a" candidate_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "796526474f254c0db0c39a41c783b7de"
I20250626 08:22:48.510979 23831 raft_consensus.cc:3053] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:22:48.511237 23831 raft_consensus.cc:738] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 796526474f254c0db0c39a41c783b7de, State: Running, Role: LEADER
I20250626 08:22:48.511993 24097 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2efe182669f347edb91170c84432fe4a" candidate_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "778031abc17a4e9393bcb0dcc2c17c73"
I20250626 08:22:48.511842 23831 consensus_queue.cc:260] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796526474f254c0db0c39a41c783b7de" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37571 } } peers { permanent_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42705 } } peers { permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 } }
I20250626 08:22:48.512439 24097 raft_consensus.cc:3058] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:48.512928 23831 raft_consensus.cc:3058] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:48.516857 24097 raft_consensus.cc:2466] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48eaaf272ca8455c8e69c89b34e0ccb4 in term 2.
I20250626 08:22:48.517284 23831 raft_consensus.cc:2466] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48eaaf272ca8455c8e69c89b34e0ccb4 in term 2.
I20250626 08:22:48.518014 23899 leader_election.cc:304] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48eaaf272ca8455c8e69c89b34e0ccb4, 778031abc17a4e9393bcb0dcc2c17c73; no voters:
I20250626 08:22:48.520051 24210 raft_consensus.cc:2802] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:22:48.521411 24210 raft_consensus.cc:695] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [term 2 LEADER]: Becoming Leader. State: Replica: 48eaaf272ca8455c8e69c89b34e0ccb4, State: Running, Role: LEADER
I20250626 08:22:48.522186 24210 consensus_queue.cc:237] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796526474f254c0db0c39a41c783b7de" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37571 } } peers { permanent_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42705 } } peers { permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 } }
I20250626 08:22:48.530485 23688 catalog_manager.cc:5582] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 reported cstate change: term changed from 1 to 2, leader changed from 796526474f254c0db0c39a41c783b7de (127.22.64.65) to 48eaaf272ca8455c8e69c89b34e0ccb4 (127.22.64.66). New cstate: current_term: 2 leader_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796526474f254c0db0c39a41c783b7de" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48eaaf272ca8455c8e69c89b34e0ccb4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:48.906648 23831 raft_consensus.cc:1273] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 2 FOLLOWER]: Refusing update from remote peer 48eaaf272ca8455c8e69c89b34e0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:48.908016 24210 consensus_queue.cc:1035] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "796526474f254c0db0c39a41c783b7de" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:48.920578 24097 raft_consensus.cc:1273] T 2efe182669f347edb91170c84432fe4a P 778031abc17a4e9393bcb0dcc2c17c73 [term 2 FOLLOWER]: Refusing update from remote peer 48eaaf272ca8455c8e69c89b34e0ccb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:48.923094 24220 consensus_queue.cc:1035] T 2efe182669f347edb91170c84432fe4a P 48eaaf272ca8455c8e69c89b34e0ccb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "778031abc17a4e9393bcb0dcc2c17c73" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 40877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:22:51.009923 23831 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2efe182669f347edb91170c84432fe4a"
dest_uuid: "796526474f254c0db0c39a41c783b7de"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35986
I20250626 08:22:51.010566 23831 raft_consensus.cc:604] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 2 FOLLOWER]: Received request to transfer leadership
I20250626 08:22:51.010900 23831 raft_consensus.cc:612] T 2efe182669f347edb91170c84432fe4a P 796526474f254c0db0c39a41c783b7de [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 08:22:52.052960 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 23747
I20250626 08:22:52.077932 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 23880
I20250626 08:22:52.107224 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 24013
I20250626 08:22:52.140667 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 23654
2025-06-26T08:22:52Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14884 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250626 08:22:52.195572 22785 test_util.cc:276] Using random seed: 1149602506
I20250626 08:22:52.199978 22785 ts_itest-base.cc:115] Starting cluster with:
I20250626 08:22:52.200186 22785 ts_itest-base.cc:116] --------------
I20250626 08:22:52.200367 22785 ts_itest-base.cc:117] 3 tablet servers
I20250626 08:22:52.200512 22785 ts_itest-base.cc:118] 3 replicas per TS
I20250626 08:22:52.200666 22785 ts_itest-base.cc:119] --------------
2025-06-26T08:22:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T08:22:52Z Disabled control of system clock
I20250626 08:22:52.242000 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:32923
--webserver_interface=127.22.64.126
--webserver_port=0
--builtin_ntp_servers=127.22.64.84:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.64.126:32923 with env {}
W20250626 08:22:52.555199 24256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:52.555773 24256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:22:52.556193 24256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:52.588687 24256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 08:22:52.589004 24256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:22:52.589210 24256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 08:22:52.589407 24256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 08:22:52.625316 24256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.64.126:32923
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:32923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.64.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:22:52.626816 24256 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:22:52.628490 24256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:52.643646 24265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:52.643707 24263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:52.643985 24262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:53.837399 24264 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:22:53.837540 24256 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:22:53.841835 24256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:22:53.845149 24256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:22:53.846632 24256 hybrid_clock.cc:648] HybridClock initialized: now 1750926173846541 us; error 77 us; skew 500 ppm
I20250626 08:22:53.847452 24256 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:53.854060 24256 webserver.cc:469] Webserver started at http://127.22.64.126:40173/ using document root <none> and password file <none>
I20250626 08:22:53.855082 24256 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:53.855279 24256 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:53.855722 24256 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:53.860410 24256 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b965dba83ed647e2bee02e932cf89e74"
format_stamp: "Formatted at 2025-06-26 08:22:53 on dist-test-slave-z1j3"
I20250626 08:22:53.861538 24256 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b965dba83ed647e2bee02e932cf89e74"
format_stamp: "Formatted at 2025-06-26 08:22:53 on dist-test-slave-z1j3"
I20250626 08:22:53.869086 24256 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250626 08:22:53.875047 24272 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:53.876204 24256 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250626 08:22:53.876562 24256 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b965dba83ed647e2bee02e932cf89e74"
format_stamp: "Formatted at 2025-06-26 08:22:53 on dist-test-slave-z1j3"
I20250626 08:22:53.876950 24256 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:53.942800 24256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:53.944324 24256 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:22:53.944778 24256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:54.019522 24256 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.126:32923
I20250626 08:22:54.019659 24323 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.126:32923 every 8 connection(s)
I20250626 08:22:54.023072 24256 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 08:22:54.028439 24324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:54.031085 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 24256
I20250626 08:22:54.031491 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 08:22:54.051527 24324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74: Bootstrap starting.
I20250626 08:22:54.058580 24324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:54.060298 24324 log.cc:826] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74: Log is configured to *not* fsync() on all Append() calls
I20250626 08:22:54.065157 24324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74: No bootstrap required, opened a new log
I20250626 08:22:54.084775 24324 raft_consensus.cc:357] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b965dba83ed647e2bee02e932cf89e74" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 32923 } }
I20250626 08:22:54.085493 24324 raft_consensus.cc:383] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:54.085701 24324 raft_consensus.cc:738] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b965dba83ed647e2bee02e932cf89e74, State: Initialized, Role: FOLLOWER
I20250626 08:22:54.086426 24324 consensus_queue.cc:260] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b965dba83ed647e2bee02e932cf89e74" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 32923 } }
I20250626 08:22:54.086952 24324 raft_consensus.cc:397] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:22:54.087194 24324 raft_consensus.cc:491] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:22:54.087457 24324 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:54.092609 24324 raft_consensus.cc:513] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b965dba83ed647e2bee02e932cf89e74" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 32923 } }
I20250626 08:22:54.093358 24324 leader_election.cc:304] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b965dba83ed647e2bee02e932cf89e74; no voters:
I20250626 08:22:54.095106 24324 leader_election.cc:290] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:22:54.095796 24329 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:54.097985 24329 raft_consensus.cc:695] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [term 1 LEADER]: Becoming Leader. State: Replica: b965dba83ed647e2bee02e932cf89e74, State: Running, Role: LEADER
I20250626 08:22:54.098845 24329 consensus_queue.cc:237] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b965dba83ed647e2bee02e932cf89e74" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 32923 } }
I20250626 08:22:54.099479 24324 sys_catalog.cc:564] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:22:54.111191 24330 sys_catalog.cc:455] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b965dba83ed647e2bee02e932cf89e74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b965dba83ed647e2bee02e932cf89e74" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 32923 } } }
I20250626 08:22:54.113684 24330 sys_catalog.cc:458] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [sys.catalog]: This master's current role is: LEADER
I20250626 08:22:54.113402 24331 sys_catalog.cc:455] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b965dba83ed647e2bee02e932cf89e74. Latest consensus state: current_term: 1 leader_uuid: "b965dba83ed647e2bee02e932cf89e74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b965dba83ed647e2bee02e932cf89e74" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 32923 } } }
I20250626 08:22:54.114638 24331 sys_catalog.cc:458] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74 [sys.catalog]: This master's current role is: LEADER
I20250626 08:22:54.117606 24339 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:22:54.128564 24339 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:22:54.143880 24339 catalog_manager.cc:1349] Generated new cluster ID: 98cebb9f6f9d46b0b7421b287cfb6d32
I20250626 08:22:54.144165 24339 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:22:54.161898 24339 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 08:22:54.164163 24339 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:22:54.181946 24339 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b965dba83ed647e2bee02e932cf89e74: Generated new TSK 0
I20250626 08:22:54.183207 24339 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 08:22:54.197582 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.65:0
--local_ip_for_outbound_sockets=127.22.64.65
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:32923
--builtin_ntp_servers=127.22.64.84:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 08:22:54.516902 24348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:54.517439 24348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:22:54.517958 24348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:54.549933 24348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:22:54.550906 24348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.65
I20250626 08:22:54.587375 24348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:32923
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.65
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:22:54.588814 24348 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:22:54.590641 24348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:54.609236 24355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:54.609630 24354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:54.611286 24357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:55.819559 24356 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1211 milliseconds
I20250626 08:22:55.819929 24348 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:22:55.824090 24348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:22:55.826999 24348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:22:55.828536 24348 hybrid_clock.cc:648] HybridClock initialized: now 1750926175828482 us; error 65 us; skew 500 ppm
I20250626 08:22:55.829424 24348 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:55.836627 24348 webserver.cc:469] Webserver started at http://127.22.64.65:44865/ using document root <none> and password file <none>
I20250626 08:22:55.837616 24348 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:55.837862 24348 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:55.838377 24348 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:55.843046 24348 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "db0b056b72cb4033841d80fd0a360bbe"
format_stamp: "Formatted at 2025-06-26 08:22:55 on dist-test-slave-z1j3"
I20250626 08:22:55.844259 24348 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "db0b056b72cb4033841d80fd0a360bbe"
format_stamp: "Formatted at 2025-06-26 08:22:55 on dist-test-slave-z1j3"
I20250626 08:22:55.852463 24348 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.001s sys 0.007s
I20250626 08:22:55.858878 24364 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:55.860141 24348 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250626 08:22:55.860484 24348 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "db0b056b72cb4033841d80fd0a360bbe"
format_stamp: "Formatted at 2025-06-26 08:22:55 on dist-test-slave-z1j3"
I20250626 08:22:55.860845 24348 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:55.920574 24348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:55.922137 24348 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:22:55.922673 24348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:55.925849 24348 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:55.930388 24348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:55.930639 24348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:55.930904 24348 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:55.931087 24348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:56.074425 24348 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.65:41279
I20250626 08:22:56.074538 24476 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.65:41279 every 8 connection(s)
I20250626 08:22:56.077117 24348 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 08:22:56.082958 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 24348
I20250626 08:22:56.083509 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 08:22:56.090035 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.66:0
--local_ip_for_outbound_sockets=127.22.64.66
--webserver_interface=127.22.64.66
--webserver_port=0
--tserver_master_addrs=127.22.64.126:32923
--builtin_ntp_servers=127.22.64.84:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:22:56.107200 24477 heartbeater.cc:344] Connected to a master server at 127.22.64.126:32923
I20250626 08:22:56.107839 24477 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:56.109428 24477 heartbeater.cc:507] Master 127.22.64.126:32923 requested a full tablet report, sending...
I20250626 08:22:56.112784 24289 ts_manager.cc:194] Registered new tserver with Master: db0b056b72cb4033841d80fd0a360bbe (127.22.64.65:41279)
I20250626 08:22:56.114907 24289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.65:42299
W20250626 08:22:56.401400 24481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:56.401899 24481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:22:56.402390 24481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:56.435108 24481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:22:56.435949 24481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.66
I20250626 08:22:56.471238 24481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.64.66
--webserver_port=0
--tserver_master_addrs=127.22.64.126:32923
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.66
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:22:56.472573 24481 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:22:56.474174 24481 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:56.491822 24490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:57.119306 24477 heartbeater.cc:499] Master 127.22.64.126:32923 was elected leader, sending a full tablet report...
W20250626 08:22:56.491847 24488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:56.491849 24487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:57.668592 24489 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:22:57.668699 24481 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:22:57.673270 24481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:22:57.675661 24481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:22:57.677073 24481 hybrid_clock.cc:648] HybridClock initialized: now 1750926177677007 us; error 73 us; skew 500 ppm
I20250626 08:22:57.678049 24481 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:57.684650 24481 webserver.cc:469] Webserver started at http://127.22.64.66:40307/ using document root <none> and password file <none>
I20250626 08:22:57.685670 24481 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:57.685910 24481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:57.686429 24481 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:57.691128 24481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "39878158f0a346e2b203dfec47c935fa"
format_stamp: "Formatted at 2025-06-26 08:22:57 on dist-test-slave-z1j3"
I20250626 08:22:57.692284 24481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "39878158f0a346e2b203dfec47c935fa"
format_stamp: "Formatted at 2025-06-26 08:22:57 on dist-test-slave-z1j3"
I20250626 08:22:57.700209 24481 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250626 08:22:57.706522 24498 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:57.707702 24481 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250626 08:22:57.708051 24481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "39878158f0a346e2b203dfec47c935fa"
format_stamp: "Formatted at 2025-06-26 08:22:57 on dist-test-slave-z1j3"
I20250626 08:22:57.708395 24481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:57.758906 24481 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:57.760416 24481 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:22:57.760908 24481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:57.763473 24481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:57.767776 24481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:57.768006 24481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:57.768262 24481 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:57.768426 24481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:57.908761 24481 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.66:45103
I20250626 08:22:57.908879 24610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.66:45103 every 8 connection(s)
I20250626 08:22:57.911440 24481 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 08:22:57.921751 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 24481
I20250626 08:22:57.922468 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 08:22:57.930423 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.67:0
--local_ip_for_outbound_sockets=127.22.64.67
--webserver_interface=127.22.64.67
--webserver_port=0
--tserver_master_addrs=127.22.64.126:32923
--builtin_ntp_servers=127.22.64.84:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:22:57.934866 24611 heartbeater.cc:344] Connected to a master server at 127.22.64.126:32923
I20250626 08:22:57.935321 24611 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:57.936401 24611 heartbeater.cc:507] Master 127.22.64.126:32923 requested a full tablet report, sending...
I20250626 08:22:57.939066 24289 ts_manager.cc:194] Registered new tserver with Master: 39878158f0a346e2b203dfec47c935fa (127.22.64.66:45103)
I20250626 08:22:57.941025 24289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.66:46169
W20250626 08:22:58.241981 24615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:22:58.242486 24615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:22:58.242939 24615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:22:58.274504 24615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:22:58.275352 24615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.67
I20250626 08:22:58.311686 24615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.64.67
--webserver_port=0
--tserver_master_addrs=127.22.64.126:32923
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.67
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:22:58.313050 24615 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:22:58.314783 24615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:58.332442 24621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:58.945436 24611 heartbeater.cc:499] Master 127.22.64.126:32923 was elected leader, sending a full tablet report...
W20250626 08:22:58.332499 24622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:58.334129 24624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:58.334663 24615 server_base.cc:1048] running on GCE node
I20250626 08:22:59.515647 24615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:22:59.518504 24615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:22:59.519981 24615 hybrid_clock.cc:648] HybridClock initialized: now 1750926179519940 us; error 60 us; skew 500 ppm
I20250626 08:22:59.520785 24615 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:59.527244 24615 webserver.cc:469] Webserver started at http://127.22.64.67:44943/ using document root <none> and password file <none>
I20250626 08:22:59.528180 24615 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:59.528384 24615 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:59.528838 24615 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:59.533414 24615 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7"
format_stamp: "Formatted at 2025-06-26 08:22:59 on dist-test-slave-z1j3"
I20250626 08:22:59.534703 24615 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7"
format_stamp: "Formatted at 2025-06-26 08:22:59 on dist-test-slave-z1j3"
I20250626 08:22:59.542024 24615 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20250626 08:22:59.548056 24631 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:59.549217 24615 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250626 08:22:59.549573 24615 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7"
format_stamp: "Formatted at 2025-06-26 08:22:59 on dist-test-slave-z1j3"
I20250626 08:22:59.549927 24615 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:59.599292 24615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:59.600816 24615 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:22:59.601328 24615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:59.604008 24615 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:59.608611 24615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:59.608824 24615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:59.609098 24615 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:59.609256 24615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:59.750164 24615 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.67:41257
I20250626 08:22:59.750293 24743 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.67:41257 every 8 connection(s)
I20250626 08:22:59.752873 24615 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 08:22:59.760502 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 24615
I20250626 08:22:59.761065 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 08:22:59.775738 24744 heartbeater.cc:344] Connected to a master server at 127.22.64.126:32923
I20250626 08:22:59.776216 24744 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:59.777272 24744 heartbeater.cc:507] Master 127.22.64.126:32923 requested a full tablet report, sending...
I20250626 08:22:59.779896 24289 ts_manager.cc:194] Registered new tserver with Master: d9bd64ab7a1f4d9a9237b12c0a7726e7 (127.22.64.67:41257)
I20250626 08:22:59.781528 24289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.67:53055
I20250626 08:22:59.782869 22785 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 08:22:59.820164 24289 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56494:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250626 08:22:59.839699 24289 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 08:22:59.906064 24679 tablet_service.cc:1468] Processing CreateTablet for tablet c43990e27d9248f3b81133a865226a7b (DEFAULT_TABLE table=TestTable [id=f8fdf4fbfdd2458dab5a6aae4338a2a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:22:59.906066 24412 tablet_service.cc:1468] Processing CreateTablet for tablet c43990e27d9248f3b81133a865226a7b (DEFAULT_TABLE table=TestTable [id=f8fdf4fbfdd2458dab5a6aae4338a2a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:22:59.906066 24546 tablet_service.cc:1468] Processing CreateTablet for tablet c43990e27d9248f3b81133a865226a7b (DEFAULT_TABLE table=TestTable [id=f8fdf4fbfdd2458dab5a6aae4338a2a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:22:59.908169 24412 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c43990e27d9248f3b81133a865226a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:59.908169 24546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c43990e27d9248f3b81133a865226a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:59.908346 24679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c43990e27d9248f3b81133a865226a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:59.944224 24764 tablet_bootstrap.cc:492] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa: Bootstrap starting.
I20250626 08:22:59.945456 24763 tablet_bootstrap.cc:492] T c43990e27d9248f3b81133a865226a7b P db0b056b72cb4033841d80fd0a360bbe: Bootstrap starting.
I20250626 08:22:59.949013 24765 tablet_bootstrap.cc:492] T c43990e27d9248f3b81133a865226a7b P d9bd64ab7a1f4d9a9237b12c0a7726e7: Bootstrap starting.
I20250626 08:22:59.953534 24763 tablet_bootstrap.cc:654] T c43990e27d9248f3b81133a865226a7b P db0b056b72cb4033841d80fd0a360bbe: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:59.956248 24763 log.cc:826] T c43990e27d9248f3b81133a865226a7b P db0b056b72cb4033841d80fd0a360bbe: Log is configured to *not* fsync() on all Append() calls
I20250626 08:22:59.959424 24765 tablet_bootstrap.cc:654] T c43990e27d9248f3b81133a865226a7b P d9bd64ab7a1f4d9a9237b12c0a7726e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:59.961793 24764 tablet_bootstrap.cc:654] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:59.963109 24765 log.cc:826] T c43990e27d9248f3b81133a865226a7b P d9bd64ab7a1f4d9a9237b12c0a7726e7: Log is configured to *not* fsync() on all Append() calls
I20250626 08:22:59.966387 24764 log.cc:826] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa: Log is configured to *not* fsync() on all Append() calls
I20250626 08:22:59.968134 24763 tablet_bootstrap.cc:492] T c43990e27d9248f3b81133a865226a7b P db0b056b72cb4033841d80fd0a360bbe: No bootstrap required, opened a new log
I20250626 08:22:59.968861 24763 ts_tablet_manager.cc:1397] T c43990e27d9248f3b81133a865226a7b P db0b056b72cb4033841d80fd0a360bbe: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.007s
I20250626 08:22:59.969827 24765 tablet_bootstrap.cc:492] T c43990e27d9248f3b81133a865226a7b P d9bd64ab7a1f4d9a9237b12c0a7726e7: No bootstrap required, opened a new log
I20250626 08:22:59.970510 24765 ts_tablet_manager.cc:1397] T c43990e27d9248f3b81133a865226a7b P d9bd64ab7a1f4d9a9237b12c0a7726e7: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.000s
I20250626 08:22:59.977998 24764 tablet_bootstrap.cc:492] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa: No bootstrap required, opened a new log
I20250626 08:22:59.978629 24764 ts_tablet_manager.cc:1397] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.014s
I20250626 08:22:59.991199 24765 raft_consensus.cc:357] T c43990e27d9248f3b81133a865226a7b P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } } peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } }
I20250626 08:22:59.991925 24765 raft_consensus.cc:383] T c43990e27d9248f3b81133a865226a7b P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:59.992206 24765 raft_consensus.cc:738] T c43990e27d9248f3b81133a865226a7b P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9bd64ab7a1f4d9a9237b12c0a7726e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:59.992960 24765 consensus_queue.cc:260] T c43990e27d9248f3b81133a865226a7b P d9bd64ab7a1f4d9a9237b12c0a7726e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } } peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } }
I20250626 08:22:59.996189 24744 heartbeater.cc:499] Master 127.22.64.126:32923 was elected leader, sending a full tablet report...
I20250626 08:22:59.997742 24765 ts_tablet_manager.cc:1428] T c43990e27d9248f3b81133a865226a7b P d9bd64ab7a1f4d9a9237b12c0a7726e7: Time spent starting tablet: real 0.027s user 0.026s sys 0.000s
I20250626 08:22:59.997979 24763 raft_consensus.cc:357] T c43990e27d9248f3b81133a865226a7b P db0b056b72cb4033841d80fd0a360bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } } peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } }
I20250626 08:22:59.998963 24763 raft_consensus.cc:383] T c43990e27d9248f3b81133a865226a7b P db0b056b72cb4033841d80fd0a360bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:59.999277 24763 raft_consensus.cc:738] T c43990e27d9248f3b81133a865226a7b P db0b056b72cb4033841d80fd0a360bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0b056b72cb4033841d80fd0a360bbe, State: Initialized, Role: FOLLOWER
I20250626 08:23:00.000319 24763 consensus_queue.cc:260] T c43990e27d9248f3b81133a865226a7b P db0b056b72cb4033841d80fd0a360bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } } peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } }
W20250626 08:23:00.007719 24745 tablet.cc:2378] T c43990e27d9248f3b81133a865226a7b P d9bd64ab7a1f4d9a9237b12c0a7726e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:23:00.007398 24764 raft_consensus.cc:357] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } } peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } }
I20250626 08:23:00.008340 24764 raft_consensus.cc:383] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:23:00.008632 24764 raft_consensus.cc:738] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39878158f0a346e2b203dfec47c935fa, State: Initialized, Role: FOLLOWER
I20250626 08:23:00.007175 24763 ts_tablet_manager.cc:1428] T c43990e27d9248f3b81133a865226a7b P db0b056b72cb4033841d80fd0a360bbe: Time spent starting tablet: real 0.038s user 0.030s sys 0.004s
I20250626 08:23:00.010093 24764 consensus_queue.cc:260] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } } peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } }
I20250626 08:23:00.014755 24764 ts_tablet_manager.cc:1428] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa: Time spent starting tablet: real 0.036s user 0.024s sys 0.011s
W20250626 08:23:00.087137 24478 tablet.cc:2378] T c43990e27d9248f3b81133a865226a7b P db0b056b72cb4033841d80fd0a360bbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 08:23:00.168994 24612 tablet.cc:2378] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:23:00.290971 24771 raft_consensus.cc:491] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:00.291440 24771 raft_consensus.cc:513] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } } peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } }
I20250626 08:23:00.293869 24771 leader_election.cc:290] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9bd64ab7a1f4d9a9237b12c0a7726e7 (127.22.64.67:41257), db0b056b72cb4033841d80fd0a360bbe (127.22.64.65:41279)
I20250626 08:23:00.307468 24699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c43990e27d9248f3b81133a865226a7b" candidate_uuid: "39878158f0a346e2b203dfec47c935fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" is_pre_election: true
I20250626 08:23:00.308254 24699 raft_consensus.cc:2466] T c43990e27d9248f3b81133a865226a7b P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39878158f0a346e2b203dfec47c935fa in term 0.
I20250626 08:23:00.308331 24432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c43990e27d9248f3b81133a865226a7b" candidate_uuid: "39878158f0a346e2b203dfec47c935fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0b056b72cb4033841d80fd0a360bbe" is_pre_election: true
I20250626 08:23:00.309170 24432 raft_consensus.cc:2466] T c43990e27d9248f3b81133a865226a7b P db0b056b72cb4033841d80fd0a360bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39878158f0a346e2b203dfec47c935fa in term 0.
I20250626 08:23:00.309350 24499 leader_election.cc:304] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39878158f0a346e2b203dfec47c935fa, d9bd64ab7a1f4d9a9237b12c0a7726e7; no voters:
I20250626 08:23:00.310124 24771 raft_consensus.cc:2802] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:23:00.310427 24771 raft_consensus.cc:491] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:00.310688 24771 raft_consensus.cc:3058] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:23:00.316062 24771 raft_consensus.cc:513] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } } peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } }
I20250626 08:23:00.317498 24771 leader_election.cc:290] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [CANDIDATE]: Term 1 election: Requested vote from peers d9bd64ab7a1f4d9a9237b12c0a7726e7 (127.22.64.67:41257), db0b056b72cb4033841d80fd0a360bbe (127.22.64.65:41279)
I20250626 08:23:00.318382 24699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c43990e27d9248f3b81133a865226a7b" candidate_uuid: "39878158f0a346e2b203dfec47c935fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7"
I20250626 08:23:00.318508 24432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c43990e27d9248f3b81133a865226a7b" candidate_uuid: "39878158f0a346e2b203dfec47c935fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0b056b72cb4033841d80fd0a360bbe"
I20250626 08:23:00.318866 24699 raft_consensus.cc:3058] T c43990e27d9248f3b81133a865226a7b P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:23:00.318987 24432 raft_consensus.cc:3058] T c43990e27d9248f3b81133a865226a7b P db0b056b72cb4033841d80fd0a360bbe [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:23:00.323269 24699 raft_consensus.cc:2466] T c43990e27d9248f3b81133a865226a7b P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39878158f0a346e2b203dfec47c935fa in term 1.
I20250626 08:23:00.323405 24432 raft_consensus.cc:2466] T c43990e27d9248f3b81133a865226a7b P db0b056b72cb4033841d80fd0a360bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39878158f0a346e2b203dfec47c935fa in term 1.
I20250626 08:23:00.324232 24499 leader_election.cc:304] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39878158f0a346e2b203dfec47c935fa, d9bd64ab7a1f4d9a9237b12c0a7726e7; no voters:
I20250626 08:23:00.324860 24771 raft_consensus.cc:2802] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:23:00.326443 24771 raft_consensus.cc:695] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [term 1 LEADER]: Becoming Leader. State: Replica: 39878158f0a346e2b203dfec47c935fa, State: Running, Role: LEADER
I20250626 08:23:00.327299 24771 consensus_queue.cc:237] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } } peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } }
I20250626 08:23:00.336251 24287 catalog_manager.cc:5582] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 39878158f0a346e2b203dfec47c935fa (127.22.64.66). New cstate: current_term: 1 leader_uuid: "39878158f0a346e2b203dfec47c935fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } health_report { overall_health: HEALTHY } } }
I20250626 08:23:00.375492 22785 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 08:23:00.379112 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver db0b056b72cb4033841d80fd0a360bbe to finish bootstrapping
I20250626 08:23:00.391400 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 39878158f0a346e2b203dfec47c935fa to finish bootstrapping
I20250626 08:23:00.403003 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d9bd64ab7a1f4d9a9237b12c0a7726e7 to finish bootstrapping
I20250626 08:23:00.416797 24287 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56494:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250626 08:23:00.418336 24287 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 08:23:00.435520 24412 tablet_service.cc:1468] Processing CreateTablet for tablet 274110fe8b0947e0bd04bc497c44869f (DEFAULT_TABLE table=TestAnotherTable [id=2401bccfab2143569e863979c12edc14]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250626 08:23:00.436014 24546 tablet_service.cc:1468] Processing CreateTablet for tablet 274110fe8b0947e0bd04bc497c44869f (DEFAULT_TABLE table=TestAnotherTable [id=2401bccfab2143569e863979c12edc14]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250626 08:23:00.436894 24412 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274110fe8b0947e0bd04bc497c44869f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:00.436847 24679 tablet_service.cc:1468] Processing CreateTablet for tablet 274110fe8b0947e0bd04bc497c44869f (DEFAULT_TABLE table=TestAnotherTable [id=2401bccfab2143569e863979c12edc14]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250626 08:23:00.437320 24546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274110fe8b0947e0bd04bc497c44869f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:00.437932 24679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274110fe8b0947e0bd04bc497c44869f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:00.450042 24765 tablet_bootstrap.cc:492] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7: Bootstrap starting.
I20250626 08:23:00.451925 24764 tablet_bootstrap.cc:492] T 274110fe8b0947e0bd04bc497c44869f P 39878158f0a346e2b203dfec47c935fa: Bootstrap starting.
I20250626 08:23:00.452569 24763 tablet_bootstrap.cc:492] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe: Bootstrap starting.
I20250626 08:23:00.456691 24765 tablet_bootstrap.cc:654] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:23:00.457351 24764 tablet_bootstrap.cc:654] T 274110fe8b0947e0bd04bc497c44869f P 39878158f0a346e2b203dfec47c935fa: Neither blocks nor log segments found. Creating new log.
I20250626 08:23:00.459141 24763 tablet_bootstrap.cc:654] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe: Neither blocks nor log segments found. Creating new log.
I20250626 08:23:00.466464 24763 tablet_bootstrap.cc:492] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe: No bootstrap required, opened a new log
I20250626 08:23:00.466928 24763 ts_tablet_manager.cc:1397] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20250626 08:23:00.467125 24765 tablet_bootstrap.cc:492] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7: No bootstrap required, opened a new log
I20250626 08:23:00.467331 24764 tablet_bootstrap.cc:492] T 274110fe8b0947e0bd04bc497c44869f P 39878158f0a346e2b203dfec47c935fa: No bootstrap required, opened a new log
I20250626 08:23:00.467545 24765 ts_tablet_manager.cc:1397] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20250626 08:23:00.467741 24764 ts_tablet_manager.cc:1397] T 274110fe8b0947e0bd04bc497c44869f P 39878158f0a346e2b203dfec47c935fa: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20250626 08:23:00.470057 24763 raft_consensus.cc:357] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } } peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } }
I20250626 08:23:00.470443 24764 raft_consensus.cc:357] T 274110fe8b0947e0bd04bc497c44869f P 39878158f0a346e2b203dfec47c935fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } } peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } }
I20250626 08:23:00.470805 24763 raft_consensus.cc:383] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:23:00.471100 24764 raft_consensus.cc:383] T 274110fe8b0947e0bd04bc497c44869f P 39878158f0a346e2b203dfec47c935fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:23:00.471107 24763 raft_consensus.cc:738] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0b056b72cb4033841d80fd0a360bbe, State: Initialized, Role: FOLLOWER
I20250626 08:23:00.471331 24764 raft_consensus.cc:738] T 274110fe8b0947e0bd04bc497c44869f P 39878158f0a346e2b203dfec47c935fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39878158f0a346e2b203dfec47c935fa, State: Initialized, Role: FOLLOWER
I20250626 08:23:00.470997 24765 raft_consensus.cc:357] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } } peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } }
I20250626 08:23:00.471720 24765 raft_consensus.cc:383] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:23:00.472079 24765 raft_consensus.cc:738] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9bd64ab7a1f4d9a9237b12c0a7726e7, State: Initialized, Role: FOLLOWER
I20250626 08:23:00.471781 24763 consensus_queue.cc:260] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } } peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } }
I20250626 08:23:00.471994 24764 consensus_queue.cc:260] T 274110fe8b0947e0bd04bc497c44869f P 39878158f0a346e2b203dfec47c935fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } } peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } }
I20250626 08:23:00.472797 24765 consensus_queue.cc:260] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } } peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } }
I20250626 08:23:00.475543 24764 ts_tablet_manager.cc:1428] T 274110fe8b0947e0bd04bc497c44869f P 39878158f0a346e2b203dfec47c935fa: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250626 08:23:00.475864 24765 ts_tablet_manager.cc:1428] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250626 08:23:00.478421 24763 ts_tablet_manager.cc:1428] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe: Time spent starting tablet: real 0.011s user 0.004s sys 0.003s
I20250626 08:23:00.520438 24770 raft_consensus.cc:491] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:00.520905 24770 raft_consensus.cc:513] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } } peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } }
I20250626 08:23:00.523371 24769 raft_consensus.cc:491] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:00.523535 24770 leader_election.cc:290] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39878158f0a346e2b203dfec47c935fa (127.22.64.66:45103), d9bd64ab7a1f4d9a9237b12c0a7726e7 (127.22.64.67:41257)
I20250626 08:23:00.523922 24769 raft_consensus.cc:513] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } } peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } }
I20250626 08:23:00.527086 24769 leader_election.cc:290] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0b056b72cb4033841d80fd0a360bbe (127.22.64.65:41279), 39878158f0a346e2b203dfec47c935fa (127.22.64.66:45103)
I20250626 08:23:00.540307 24566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274110fe8b0947e0bd04bc497c44869f" candidate_uuid: "db0b056b72cb4033841d80fd0a360bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39878158f0a346e2b203dfec47c935fa" is_pre_election: true
I20250626 08:23:00.540938 24699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274110fe8b0947e0bd04bc497c44869f" candidate_uuid: "db0b056b72cb4033841d80fd0a360bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" is_pre_election: true
I20250626 08:23:00.541162 24566 raft_consensus.cc:2466] T 274110fe8b0947e0bd04bc497c44869f P 39878158f0a346e2b203dfec47c935fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0b056b72cb4033841d80fd0a360bbe in term 0.
I20250626 08:23:00.541541 24699 raft_consensus.cc:2466] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0b056b72cb4033841d80fd0a360bbe in term 0.
I20250626 08:23:00.542747 24365 leader_election.cc:304] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9bd64ab7a1f4d9a9237b12c0a7726e7, db0b056b72cb4033841d80fd0a360bbe; no voters:
I20250626 08:23:00.543996 24770 raft_consensus.cc:2802] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:23:00.544471 24770 raft_consensus.cc:491] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:00.544855 24770 raft_consensus.cc:3058] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:23:00.551865 24432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274110fe8b0947e0bd04bc497c44869f" candidate_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0b056b72cb4033841d80fd0a360bbe" is_pre_election: true
I20250626 08:23:00.552477 24566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274110fe8b0947e0bd04bc497c44869f" candidate_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39878158f0a346e2b203dfec47c935fa" is_pre_election: true
I20250626 08:23:00.553086 24566 raft_consensus.cc:2466] T 274110fe8b0947e0bd04bc497c44869f P 39878158f0a346e2b203dfec47c935fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9bd64ab7a1f4d9a9237b12c0a7726e7 in term 0.
I20250626 08:23:00.552827 24770 raft_consensus.cc:513] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } } peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } }
I20250626 08:23:00.553969 24432 raft_consensus.cc:2391] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9bd64ab7a1f4d9a9237b12c0a7726e7 in current term 1: Already voted for candidate db0b056b72cb4033841d80fd0a360bbe in this term.
I20250626 08:23:00.554390 24635 leader_election.cc:304] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39878158f0a346e2b203dfec47c935fa, d9bd64ab7a1f4d9a9237b12c0a7726e7; no voters:
I20250626 08:23:00.555132 24769 raft_consensus.cc:2802] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:23:00.555398 24769 raft_consensus.cc:491] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:00.555661 24769 raft_consensus.cc:3058] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:23:00.556174 24566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274110fe8b0947e0bd04bc497c44869f" candidate_uuid: "db0b056b72cb4033841d80fd0a360bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39878158f0a346e2b203dfec47c935fa"
I20250626 08:23:00.556727 24770 leader_election.cc:290] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [CANDIDATE]: Term 1 election: Requested vote from peers 39878158f0a346e2b203dfec47c935fa (127.22.64.66:45103), d9bd64ab7a1f4d9a9237b12c0a7726e7 (127.22.64.67:41257)
I20250626 08:23:00.556717 24566 raft_consensus.cc:3058] T 274110fe8b0947e0bd04bc497c44869f P 39878158f0a346e2b203dfec47c935fa [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:23:00.557943 24699 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274110fe8b0947e0bd04bc497c44869f" candidate_uuid: "db0b056b72cb4033841d80fd0a360bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7"
I20250626 08:23:00.562783 24769 raft_consensus.cc:513] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } } peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } }
I20250626 08:23:00.564028 24699 raft_consensus.cc:2391] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate db0b056b72cb4033841d80fd0a360bbe in current term 1: Already voted for candidate d9bd64ab7a1f4d9a9237b12c0a7726e7 in this term.
I20250626 08:23:00.563894 24566 raft_consensus.cc:2466] T 274110fe8b0947e0bd04bc497c44869f P 39878158f0a346e2b203dfec47c935fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0b056b72cb4033841d80fd0a360bbe in term 1.
I20250626 08:23:00.565023 24769 leader_election.cc:290] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [CANDIDATE]: Term 1 election: Requested vote from peers db0b056b72cb4033841d80fd0a360bbe (127.22.64.65:41279), 39878158f0a346e2b203dfec47c935fa (127.22.64.66:45103)
I20250626 08:23:00.565122 24368 leader_election.cc:304] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39878158f0a346e2b203dfec47c935fa, db0b056b72cb4033841d80fd0a360bbe; no voters:
I20250626 08:23:00.566191 24770 raft_consensus.cc:2802] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:23:00.566684 24566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274110fe8b0947e0bd04bc497c44869f" candidate_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39878158f0a346e2b203dfec47c935fa"
I20250626 08:23:00.567282 24566 raft_consensus.cc:2391] T 274110fe8b0947e0bd04bc497c44869f P 39878158f0a346e2b203dfec47c935fa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d9bd64ab7a1f4d9a9237b12c0a7726e7 in current term 1: Already voted for candidate db0b056b72cb4033841d80fd0a360bbe in this term.
I20250626 08:23:00.570639 24770 raft_consensus.cc:695] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [term 1 LEADER]: Becoming Leader. State: Replica: db0b056b72cb4033841d80fd0a360bbe, State: Running, Role: LEADER
I20250626 08:23:00.570683 24432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274110fe8b0947e0bd04bc497c44869f" candidate_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0b056b72cb4033841d80fd0a360bbe"
I20250626 08:23:00.571702 24770 consensus_queue.cc:237] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } } peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } }
I20250626 08:23:00.572118 24634 leader_election.cc:304] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9bd64ab7a1f4d9a9237b12c0a7726e7; no voters: 39878158f0a346e2b203dfec47c935fa, db0b056b72cb4033841d80fd0a360bbe
I20250626 08:23:00.573036 24769 raft_consensus.cc:2747] T 274110fe8b0947e0bd04bc497c44869f P d9bd64ab7a1f4d9a9237b12c0a7726e7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 08:23:00.581395 24288 catalog_manager.cc:5582] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe reported cstate change: term changed from 0 to 1, leader changed from <none> to db0b056b72cb4033841d80fd0a360bbe (127.22.64.65). New cstate: current_term: 1 leader_uuid: "db0b056b72cb4033841d80fd0a360bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:00.874795 24771 consensus_queue.cc:1035] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:23:00.895577 24791 consensus_queue.cc:1035] T c43990e27d9248f3b81133a865226a7b P 39878158f0a346e2b203dfec47c935fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0b056b72cb4033841d80fd0a360bbe" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 41279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250626 08:23:00.943482 24787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:00.944360 24787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:00.976011 24787 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250626 08:23:01.065577 24770 consensus_queue.cc:1035] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9bd64ab7a1f4d9a9237b12c0a7726e7" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 41257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:23:01.076545 24785 consensus_queue.cc:1035] T 274110fe8b0947e0bd04bc497c44869f P db0b056b72cb4033841d80fd0a360bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "39878158f0a346e2b203dfec47c935fa" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 45103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 08:23:02.345726 24787 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.324s user 0.478s sys 0.830s
W20250626 08:23:02.346024 24787 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.324s user 0.478s sys 0.830s
W20250626 08:23:03.736351 24811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:03.737740 24811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:03.770946 24811 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 08:23:05.112131 24811 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.294s user 0.530s sys 0.762s
W20250626 08:23:05.112589 24811 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.294s user 0.531s sys 0.763s
W20250626 08:23:06.517827 24825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:06.518553 24825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:06.553264 24825 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 08:23:07.895484 24825 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.295s user 0.510s sys 0.783s
W20250626 08:23:07.895817 24825 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.296s user 0.510s sys 0.783s
W20250626 08:23:09.292748 24844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:09.293380 24844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:09.324889 24844 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 08:23:10.679804 24844 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.312s user 0.506s sys 0.804s
W20250626 08:23:10.680133 24844 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.313s user 0.506s sys 0.804s
I20250626 08:23:11.763622 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 24348
I20250626 08:23:11.795331 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 24481
I20250626 08:23:11.825301 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 24615
I20250626 08:23:11.853583 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 24256
2025-06-26T08:23:11Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (19721 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250626 08:23:11.917344 22785 test_util.cc:276] Using random seed: 1169324293
I20250626 08:23:11.921811 22785 ts_itest-base.cc:115] Starting cluster with:
I20250626 08:23:11.921989 22785 ts_itest-base.cc:116] --------------
I20250626 08:23:11.922156 22785 ts_itest-base.cc:117] 3 tablet servers
I20250626 08:23:11.922343 22785 ts_itest-base.cc:118] 3 replicas per TS
I20250626 08:23:11.922509 22785 ts_itest-base.cc:119] --------------
2025-06-26T08:23:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T08:23:11Z Disabled control of system clock
I20250626 08:23:11.963120 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:35299
--webserver_interface=127.22.64.126
--webserver_port=0
--builtin_ntp_servers=127.22.64.84:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.64.126:35299
--superuser_acl=no-such-user with env {}
W20250626 08:23:12.271759 24866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:12.272336 24866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:12.272799 24866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:12.305146 24866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 08:23:12.305454 24866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:12.305671 24866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 08:23:12.305869 24866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 08:23:12.341567 24866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.64.126:35299
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:35299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.64.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:12.342898 24866 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:12.344568 24866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:12.360030 24873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:12.360098 24875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:12.360239 24872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:12.361529 24866 server_base.cc:1048] running on GCE node
I20250626 08:23:13.555509 24866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:13.558516 24866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:13.559926 24866 hybrid_clock.cc:648] HybridClock initialized: now 1750926193559887 us; error 47 us; skew 500 ppm
I20250626 08:23:13.560810 24866 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:13.574028 24866 webserver.cc:469] Webserver started at http://127.22.64.126:40343/ using document root <none> and password file <none>
I20250626 08:23:13.575340 24866 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:13.575600 24866 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:13.576140 24866 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:23:13.580811 24866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "af32bd19cad64ccd9125a4ccd5ecfc7e"
format_stamp: "Formatted at 2025-06-26 08:23:13 on dist-test-slave-z1j3"
I20250626 08:23:13.582011 24866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "af32bd19cad64ccd9125a4ccd5ecfc7e"
format_stamp: "Formatted at 2025-06-26 08:23:13 on dist-test-slave-z1j3"
I20250626 08:23:13.589694 24866 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250626 08:23:13.595757 24882 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:13.596984 24866 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250626 08:23:13.597321 24866 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
uuid: "af32bd19cad64ccd9125a4ccd5ecfc7e"
format_stamp: "Formatted at 2025-06-26 08:23:13 on dist-test-slave-z1j3"
I20250626 08:23:13.597662 24866 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:13.668233 24866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:13.669780 24866 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:13.670271 24866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:13.742803 24866 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.126:35299
I20250626 08:23:13.742882 24933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.126:35299 every 8 connection(s)
I20250626 08:23:13.745580 24866 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 08:23:13.750830 24934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:13.751623 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 24866
I20250626 08:23:13.752131 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 08:23:13.777706 24934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e: Bootstrap starting.
I20250626 08:23:13.783578 24934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e: Neither blocks nor log segments found. Creating new log.
I20250626 08:23:13.785398 24934 log.cc:826] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:13.790751 24934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e: No bootstrap required, opened a new log
I20250626 08:23:13.813076 24934 raft_consensus.cc:357] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af32bd19cad64ccd9125a4ccd5ecfc7e" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 35299 } }
I20250626 08:23:13.813805 24934 raft_consensus.cc:383] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:23:13.814010 24934 raft_consensus.cc:738] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af32bd19cad64ccd9125a4ccd5ecfc7e, State: Initialized, Role: FOLLOWER
I20250626 08:23:13.814718 24934 consensus_queue.cc:260] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af32bd19cad64ccd9125a4ccd5ecfc7e" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 35299 } }
I20250626 08:23:13.815210 24934 raft_consensus.cc:397] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:23:13.815443 24934 raft_consensus.cc:491] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:23:13.815740 24934 raft_consensus.cc:3058] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:23:13.820371 24934 raft_consensus.cc:513] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af32bd19cad64ccd9125a4ccd5ecfc7e" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 35299 } }
I20250626 08:23:13.821250 24934 leader_election.cc:304] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af32bd19cad64ccd9125a4ccd5ecfc7e; no voters:
I20250626 08:23:13.823113 24934 leader_election.cc:290] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:23:13.823855 24939 raft_consensus.cc:2802] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:23:13.826367 24939 raft_consensus.cc:695] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [term 1 LEADER]: Becoming Leader. State: Replica: af32bd19cad64ccd9125a4ccd5ecfc7e, State: Running, Role: LEADER
I20250626 08:23:13.826900 24934 sys_catalog.cc:564] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:23:13.827206 24939 consensus_queue.cc:237] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af32bd19cad64ccd9125a4ccd5ecfc7e" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 35299 } }
I20250626 08:23:13.839064 24940 sys_catalog.cc:455] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af32bd19cad64ccd9125a4ccd5ecfc7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af32bd19cad64ccd9125a4ccd5ecfc7e" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 35299 } } }
I20250626 08:23:13.841140 24940 sys_catalog.cc:458] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [sys.catalog]: This master's current role is: LEADER
I20250626 08:23:13.840451 24941 sys_catalog.cc:455] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader af32bd19cad64ccd9125a4ccd5ecfc7e. Latest consensus state: current_term: 1 leader_uuid: "af32bd19cad64ccd9125a4ccd5ecfc7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af32bd19cad64ccd9125a4ccd5ecfc7e" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 35299 } } }
I20250626 08:23:13.841537 24941 sys_catalog.cc:458] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e [sys.catalog]: This master's current role is: LEADER
I20250626 08:23:13.844322 24949 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:23:13.858207 24949 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:23:13.874676 24949 catalog_manager.cc:1349] Generated new cluster ID: 83d47ab224c4436baddd9d4c5fb1ddc1
I20250626 08:23:13.875110 24949 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:23:13.915557 24949 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 08:23:13.917579 24949 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:23:13.954928 24949 catalog_manager.cc:5955] T 00000000000000000000000000000000 P af32bd19cad64ccd9125a4ccd5ecfc7e: Generated new TSK 0
I20250626 08:23:13.956115 24949 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 08:23:13.969619 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.65:0
--local_ip_for_outbound_sockets=127.22.64.65
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:35299
--builtin_ntp_servers=127.22.64.84:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 08:23:14.286968 24958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:14.287515 24958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:14.288018 24958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:14.320207 24958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:14.321118 24958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.65
I20250626 08:23:14.358088 24958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:35299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.65
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:14.359539 24958 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:14.361344 24958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:14.380373 24965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:14.380399 24964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:14.384673 24967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:15.590251 24966 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:23:15.590369 24958 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:23:15.591727 24958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:15.594530 24958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:15.595968 24958 hybrid_clock.cc:648] HybridClock initialized: now 1750926195595907 us; error 81 us; skew 500 ppm
I20250626 08:23:15.596804 24958 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:15.604303 24958 webserver.cc:469] Webserver started at http://127.22.64.65:38483/ using document root <none> and password file <none>
I20250626 08:23:15.605245 24958 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:15.605453 24958 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:15.605954 24958 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:23:15.610618 24958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f20fec5f1a084e20979923ec0837c95c"
format_stamp: "Formatted at 2025-06-26 08:23:15 on dist-test-slave-z1j3"
I20250626 08:23:15.611799 24958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f20fec5f1a084e20979923ec0837c95c"
format_stamp: "Formatted at 2025-06-26 08:23:15 on dist-test-slave-z1j3"
I20250626 08:23:15.619042 24958 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250626 08:23:15.624544 24974 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:15.625612 24958 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250626 08:23:15.625946 24958 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f20fec5f1a084e20979923ec0837c95c"
format_stamp: "Formatted at 2025-06-26 08:23:15 on dist-test-slave-z1j3"
I20250626 08:23:15.626292 24958 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:15.685354 24958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:15.686935 24958 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:15.687386 24958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:15.689883 24958 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:23:15.694032 24958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:23:15.694350 24958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250626 08:23:15.694628 24958 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:23:15.694800 24958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:15.835387 24958 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.65:40781
I20250626 08:23:15.835492 25086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.65:40781 every 8 connection(s)
I20250626 08:23:15.838028 24958 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 08:23:15.848479 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 24958
I20250626 08:23:15.849021 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 08:23:15.855995 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.66:0
--local_ip_for_outbound_sockets=127.22.64.66
--webserver_interface=127.22.64.66
--webserver_port=0
--tserver_master_addrs=127.22.64.126:35299
--builtin_ntp_servers=127.22.64.84:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:23:15.861119 25087 heartbeater.cc:344] Connected to a master server at 127.22.64.126:35299
I20250626 08:23:15.861687 25087 heartbeater.cc:461] Registering TS with master...
I20250626 08:23:15.863111 25087 heartbeater.cc:507] Master 127.22.64.126:35299 requested a full tablet report, sending...
I20250626 08:23:15.866456 24899 ts_manager.cc:194] Registered new tserver with Master: f20fec5f1a084e20979923ec0837c95c (127.22.64.65:40781)
I20250626 08:23:15.869361 24899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.65:44879
W20250626 08:23:16.161962 25091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:16.162513 25091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:16.162985 25091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:16.195024 25091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:16.195869 25091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.66
I20250626 08:23:16.231696 25091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.64.66
--webserver_port=0
--tserver_master_addrs=127.22.64.126:35299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.66
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:16.233048 25091 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:16.234848 25091 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:16.252691 25097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:16.873402 25087 heartbeater.cc:499] Master 127.22.64.126:35299 was elected leader, sending a full tablet report...
W20250626 08:23:16.252774 25100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:16.255343 25091 server_base.cc:1048] running on GCE node
W20250626 08:23:16.252794 25098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:17.425704 25091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:17.428053 25091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:17.429446 25091 hybrid_clock.cc:648] HybridClock initialized: now 1750926197429409 us; error 63 us; skew 500 ppm
I20250626 08:23:17.430305 25091 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:17.436697 25091 webserver.cc:469] Webserver started at http://127.22.64.66:34965/ using document root <none> and password file <none>
I20250626 08:23:17.437667 25091 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:17.437899 25091 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:17.438403 25091 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:23:17.443310 25091 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ce91f820098f48c089495f536a430451"
format_stamp: "Formatted at 2025-06-26 08:23:17 on dist-test-slave-z1j3"
I20250626 08:23:17.444500 25091 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ce91f820098f48c089495f536a430451"
format_stamp: "Formatted at 2025-06-26 08:23:17 on dist-test-slave-z1j3"
I20250626 08:23:17.451812 25091 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250626 08:23:17.457741 25107 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:17.458845 25091 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250626 08:23:17.459175 25091 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ce91f820098f48c089495f536a430451"
format_stamp: "Formatted at 2025-06-26 08:23:17 on dist-test-slave-z1j3"
I20250626 08:23:17.459523 25091 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:17.522325 25091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:17.523860 25091 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:17.524327 25091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:17.526906 25091 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:23:17.531286 25091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:23:17.531503 25091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:17.531795 25091 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:23:17.531945 25091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:17.666469 25091 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.66:32781
I20250626 08:23:17.666592 25226 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.66:32781 every 8 connection(s)
I20250626 08:23:17.669024 25091 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 08:23:17.678854 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 25091
I20250626 08:23:17.679406 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 08:23:17.686717 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.67:0
--local_ip_for_outbound_sockets=127.22.64.67
--webserver_interface=127.22.64.67
--webserver_port=0
--tserver_master_addrs=127.22.64.126:35299
--builtin_ntp_servers=127.22.64.84:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:23:17.691675 25227 heartbeater.cc:344] Connected to a master server at 127.22.64.126:35299
I20250626 08:23:17.692262 25227 heartbeater.cc:461] Registering TS with master...
I20250626 08:23:17.693696 25227 heartbeater.cc:507] Master 127.22.64.126:35299 requested a full tablet report, sending...
I20250626 08:23:17.696555 24899 ts_manager.cc:194] Registered new tserver with Master: ce91f820098f48c089495f536a430451 (127.22.64.66:32781)
I20250626 08:23:17.698551 24899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.66:47371
W20250626 08:23:18.009445 25231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:18.009943 25231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:18.010478 25231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:18.042147 25231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:18.043146 25231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.67
I20250626 08:23:18.078739 25231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.64.67
--webserver_port=0
--tserver_master_addrs=127.22.64.126:35299
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.67
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:18.080065 25231 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:18.081683 25231 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:18.099681 25237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:18.702420 25227 heartbeater.cc:499] Master 127.22.64.126:35299 was elected leader, sending a full tablet report...
W20250626 08:23:18.099956 25238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:18.100203 25240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:19.264824 25239 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:23:19.264966 25231 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:23:19.269340 25231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:19.271706 25231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:19.273087 25231 hybrid_clock.cc:648] HybridClock initialized: now 1750926199273048 us; error 56 us; skew 500 ppm
I20250626 08:23:19.273910 25231 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:19.286646 25231 webserver.cc:469] Webserver started at http://127.22.64.67:32843/ using document root <none> and password file <none>
I20250626 08:23:19.287704 25231 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:19.287951 25231 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:19.288407 25231 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:23:19.295419 25231 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f1175bc70fda4a3092a6a69cce16d257"
format_stamp: "Formatted at 2025-06-26 08:23:19 on dist-test-slave-z1j3"
I20250626 08:23:19.296618 25231 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f1175bc70fda4a3092a6a69cce16d257"
format_stamp: "Formatted at 2025-06-26 08:23:19 on dist-test-slave-z1j3"
I20250626 08:23:19.304010 25231 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20250626 08:23:19.309921 25247 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:19.311059 25231 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250626 08:23:19.311388 25231 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f1175bc70fda4a3092a6a69cce16d257"
format_stamp: "Formatted at 2025-06-26 08:23:19 on dist-test-slave-z1j3"
I20250626 08:23:19.311722 25231 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:19.363330 25231 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:19.364888 25231 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:19.365340 25231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:19.367888 25231 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:23:19.372051 25231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:23:19.372272 25231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:19.372525 25231 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:23:19.372686 25231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:19.512229 25231 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.67:34707
I20250626 08:23:19.512346 25359 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.67:34707 every 8 connection(s)
I20250626 08:23:19.514863 25231 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 08:23:19.518172 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 25231
I20250626 08:23:19.518702 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 08:23:19.537192 25360 heartbeater.cc:344] Connected to a master server at 127.22.64.126:35299
I20250626 08:23:19.537672 25360 heartbeater.cc:461] Registering TS with master...
I20250626 08:23:19.538986 25360 heartbeater.cc:507] Master 127.22.64.126:35299 requested a full tablet report, sending...
I20250626 08:23:19.541235 24899 ts_manager.cc:194] Registered new tserver with Master: f1175bc70fda4a3092a6a69cce16d257 (127.22.64.67:34707)
I20250626 08:23:19.542526 24899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.67:42177
I20250626 08:23:19.554672 22785 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 08:23:19.590978 24899 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53438:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250626 08:23:19.610596 24899 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 08:23:19.662307 25162 tablet_service.cc:1468] Processing CreateTablet for tablet d031cf4021e84b62a936b84505e12c23 (DEFAULT_TABLE table=TestTable [id=5e9dee24a78d432da7b0ae87ad37a1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:23:19.664405 25162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d031cf4021e84b62a936b84505e12c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:19.664235 25295 tablet_service.cc:1468] Processing CreateTablet for tablet d031cf4021e84b62a936b84505e12c23 (DEFAULT_TABLE table=TestTable [id=5e9dee24a78d432da7b0ae87ad37a1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:23:19.666270 25295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d031cf4021e84b62a936b84505e12c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:19.665993 25022 tablet_service.cc:1468] Processing CreateTablet for tablet d031cf4021e84b62a936b84505e12c23 (DEFAULT_TABLE table=TestTable [id=5e9dee24a78d432da7b0ae87ad37a1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:23:19.667980 25022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d031cf4021e84b62a936b84505e12c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:19.700615 25380 tablet_bootstrap.cc:492] T d031cf4021e84b62a936b84505e12c23 P f1175bc70fda4a3092a6a69cce16d257: Bootstrap starting.
I20250626 08:23:19.703781 25379 tablet_bootstrap.cc:492] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451: Bootstrap starting.
I20250626 08:23:19.710019 25380 tablet_bootstrap.cc:654] T d031cf4021e84b62a936b84505e12c23 P f1175bc70fda4a3092a6a69cce16d257: Neither blocks nor log segments found. Creating new log.
I20250626 08:23:19.712455 25379 tablet_bootstrap.cc:654] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451: Neither blocks nor log segments found. Creating new log.
I20250626 08:23:19.713301 25380 log.cc:826] T d031cf4021e84b62a936b84505e12c23 P f1175bc70fda4a3092a6a69cce16d257: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:19.714613 25381 tablet_bootstrap.cc:492] T d031cf4021e84b62a936b84505e12c23 P f20fec5f1a084e20979923ec0837c95c: Bootstrap starting.
I20250626 08:23:19.715325 25379 log.cc:826] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:19.723659 25381 tablet_bootstrap.cc:654] T d031cf4021e84b62a936b84505e12c23 P f20fec5f1a084e20979923ec0837c95c: Neither blocks nor log segments found. Creating new log.
I20250626 08:23:19.724076 25380 tablet_bootstrap.cc:492] T d031cf4021e84b62a936b84505e12c23 P f1175bc70fda4a3092a6a69cce16d257: No bootstrap required, opened a new log
I20250626 08:23:19.724359 25379 tablet_bootstrap.cc:492] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451: No bootstrap required, opened a new log
I20250626 08:23:19.724704 25380 ts_tablet_manager.cc:1397] T d031cf4021e84b62a936b84505e12c23 P f1175bc70fda4a3092a6a69cce16d257: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.009s
I20250626 08:23:19.724867 25379 ts_tablet_manager.cc:1397] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.005s
I20250626 08:23:19.725770 25381 log.cc:826] T d031cf4021e84b62a936b84505e12c23 P f20fec5f1a084e20979923ec0837c95c: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:19.731192 25381 tablet_bootstrap.cc:492] T d031cf4021e84b62a936b84505e12c23 P f20fec5f1a084e20979923ec0837c95c: No bootstrap required, opened a new log
I20250626 08:23:19.731671 25381 ts_tablet_manager.cc:1397] T d031cf4021e84b62a936b84505e12c23 P f20fec5f1a084e20979923ec0837c95c: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20250626 08:23:19.753258 25379 raft_consensus.cc:357] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1175bc70fda4a3092a6a69cce16d257" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34707 } } peers { permanent_uuid: "ce91f820098f48c089495f536a430451" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 32781 } } peers { permanent_uuid: "f20fec5f1a084e20979923ec0837c95c" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 40781 } }
I20250626 08:23:19.753258 25380 raft_consensus.cc:357] T d031cf4021e84b62a936b84505e12c23 P f1175bc70fda4a3092a6a69cce16d257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1175bc70fda4a3092a6a69cce16d257" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34707 } } peers { permanent_uuid: "ce91f820098f48c089495f536a430451" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 32781 } } peers { permanent_uuid: "f20fec5f1a084e20979923ec0837c95c" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 40781 } }
I20250626 08:23:19.754304 25380 raft_consensus.cc:383] T d031cf4021e84b62a936b84505e12c23 P f1175bc70fda4a3092a6a69cce16d257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:23:19.754364 25379 raft_consensus.cc:383] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:23:19.754652 25379 raft_consensus.cc:738] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce91f820098f48c089495f536a430451, State: Initialized, Role: FOLLOWER
I20250626 08:23:19.754688 25380 raft_consensus.cc:738] T d031cf4021e84b62a936b84505e12c23 P f1175bc70fda4a3092a6a69cce16d257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1175bc70fda4a3092a6a69cce16d257, State: Initialized, Role: FOLLOWER
I20250626 08:23:19.755640 25380 consensus_queue.cc:260] T d031cf4021e84b62a936b84505e12c23 P f1175bc70fda4a3092a6a69cce16d257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1175bc70fda4a3092a6a69cce16d257" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34707 } } peers { permanent_uuid: "ce91f820098f48c089495f536a430451" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 32781 } } peers { permanent_uuid: "f20fec5f1a084e20979923ec0837c95c" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 40781 } }
I20250626 08:23:19.755643 25379 consensus_queue.cc:260] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1175bc70fda4a3092a6a69cce16d257" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34707 } } peers { permanent_uuid: "ce91f820098f48c089495f536a430451" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 32781 } } peers { permanent_uuid: "f20fec5f1a084e20979923ec0837c95c" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 40781 } }
I20250626 08:23:19.756865 25381 raft_consensus.cc:357] T d031cf4021e84b62a936b84505e12c23 P f20fec5f1a084e20979923ec0837c95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1175bc70fda4a3092a6a69cce16d257" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34707 } } peers { permanent_uuid: "ce91f820098f48c089495f536a430451" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 32781 } } peers { permanent_uuid: "f20fec5f1a084e20979923ec0837c95c" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 40781 } }
I20250626 08:23:19.757850 25381 raft_consensus.cc:383] T d031cf4021e84b62a936b84505e12c23 P f20fec5f1a084e20979923ec0837c95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:23:19.758195 25381 raft_consensus.cc:738] T d031cf4021e84b62a936b84505e12c23 P f20fec5f1a084e20979923ec0837c95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20fec5f1a084e20979923ec0837c95c, State: Initialized, Role: FOLLOWER
I20250626 08:23:19.759327 25381 consensus_queue.cc:260] T d031cf4021e84b62a936b84505e12c23 P f20fec5f1a084e20979923ec0837c95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1175bc70fda4a3092a6a69cce16d257" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34707 } } peers { permanent_uuid: "ce91f820098f48c089495f536a430451" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 32781 } } peers { permanent_uuid: "f20fec5f1a084e20979923ec0837c95c" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 40781 } }
I20250626 08:23:19.767153 25379 ts_tablet_manager.cc:1428] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451: Time spent starting tablet: real 0.042s user 0.026s sys 0.016s
I20250626 08:23:19.770278 25380 ts_tablet_manager.cc:1428] T d031cf4021e84b62a936b84505e12c23 P f1175bc70fda4a3092a6a69cce16d257: Time spent starting tablet: real 0.045s user 0.029s sys 0.019s
I20250626 08:23:19.772090 25360 heartbeater.cc:499] Master 127.22.64.126:35299 was elected leader, sending a full tablet report...
I20250626 08:23:19.776780 25381 ts_tablet_manager.cc:1428] T d031cf4021e84b62a936b84505e12c23 P f20fec5f1a084e20979923ec0837c95c: Time spent starting tablet: real 0.045s user 0.029s sys 0.015s
W20250626 08:23:19.846587 25088 tablet.cc:2378] T d031cf4021e84b62a936b84505e12c23 P f20fec5f1a084e20979923ec0837c95c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 08:23:19.926268 25228 tablet.cc:2378] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:23:20.009932 25386 raft_consensus.cc:491] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:20.010501 25386 raft_consensus.cc:513] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1175bc70fda4a3092a6a69cce16d257" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34707 } } peers { permanent_uuid: "ce91f820098f48c089495f536a430451" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 32781 } } peers { permanent_uuid: "f20fec5f1a084e20979923ec0837c95c" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 40781 } }
I20250626 08:23:20.012974 25386 leader_election.cc:290] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1175bc70fda4a3092a6a69cce16d257 (127.22.64.67:34707), f20fec5f1a084e20979923ec0837c95c (127.22.64.65:40781)
W20250626 08:23:20.020051 25361 tablet.cc:2378] T d031cf4021e84b62a936b84505e12c23 P f1175bc70fda4a3092a6a69cce16d257: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:23:20.026584 25315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d031cf4021e84b62a936b84505e12c23" candidate_uuid: "ce91f820098f48c089495f536a430451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1175bc70fda4a3092a6a69cce16d257" is_pre_election: true
I20250626 08:23:20.026575 25042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d031cf4021e84b62a936b84505e12c23" candidate_uuid: "ce91f820098f48c089495f536a430451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20fec5f1a084e20979923ec0837c95c" is_pre_election: true
I20250626 08:23:20.027297 25315 raft_consensus.cc:2466] T d031cf4021e84b62a936b84505e12c23 P f1175bc70fda4a3092a6a69cce16d257 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce91f820098f48c089495f536a430451 in term 0.
I20250626 08:23:20.027303 25042 raft_consensus.cc:2466] T d031cf4021e84b62a936b84505e12c23 P f20fec5f1a084e20979923ec0837c95c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce91f820098f48c089495f536a430451 in term 0.
I20250626 08:23:20.028676 25110 leader_election.cc:304] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce91f820098f48c089495f536a430451, f20fec5f1a084e20979923ec0837c95c; no voters:
I20250626 08:23:20.029496 25386 raft_consensus.cc:2802] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:23:20.029785 25386 raft_consensus.cc:491] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:20.029999 25386 raft_consensus.cc:3058] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:23:20.035027 25386 raft_consensus.cc:513] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1175bc70fda4a3092a6a69cce16d257" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34707 } } peers { permanent_uuid: "ce91f820098f48c089495f536a430451" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 32781 } } peers { permanent_uuid: "f20fec5f1a084e20979923ec0837c95c" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 40781 } }
I20250626 08:23:20.036502 25386 leader_election.cc:290] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [CANDIDATE]: Term 1 election: Requested vote from peers f1175bc70fda4a3092a6a69cce16d257 (127.22.64.67:34707), f20fec5f1a084e20979923ec0837c95c (127.22.64.65:40781)
I20250626 08:23:20.037629 25315 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d031cf4021e84b62a936b84505e12c23" candidate_uuid: "ce91f820098f48c089495f536a430451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1175bc70fda4a3092a6a69cce16d257"
I20250626 08:23:20.037627 25042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d031cf4021e84b62a936b84505e12c23" candidate_uuid: "ce91f820098f48c089495f536a430451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20fec5f1a084e20979923ec0837c95c"
I20250626 08:23:20.038079 25315 raft_consensus.cc:3058] T d031cf4021e84b62a936b84505e12c23 P f1175bc70fda4a3092a6a69cce16d257 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:23:20.038085 25042 raft_consensus.cc:3058] T d031cf4021e84b62a936b84505e12c23 P f20fec5f1a084e20979923ec0837c95c [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:23:20.042834 25315 raft_consensus.cc:2466] T d031cf4021e84b62a936b84505e12c23 P f1175bc70fda4a3092a6a69cce16d257 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce91f820098f48c089495f536a430451 in term 1.
I20250626 08:23:20.042836 25042 raft_consensus.cc:2466] T d031cf4021e84b62a936b84505e12c23 P f20fec5f1a084e20979923ec0837c95c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce91f820098f48c089495f536a430451 in term 1.
I20250626 08:23:20.043915 25110 leader_election.cc:304] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce91f820098f48c089495f536a430451, f20fec5f1a084e20979923ec0837c95c; no voters:
I20250626 08:23:20.044588 25386 raft_consensus.cc:2802] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:23:20.046326 25386 raft_consensus.cc:695] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [term 1 LEADER]: Becoming Leader. State: Replica: ce91f820098f48c089495f536a430451, State: Running, Role: LEADER
I20250626 08:23:20.047101 25386 consensus_queue.cc:237] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1175bc70fda4a3092a6a69cce16d257" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34707 } } peers { permanent_uuid: "ce91f820098f48c089495f536a430451" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 32781 } } peers { permanent_uuid: "f20fec5f1a084e20979923ec0837c95c" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 40781 } }
I20250626 08:23:20.059036 24896 catalog_manager.cc:5582] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce91f820098f48c089495f536a430451 (127.22.64.66). New cstate: current_term: 1 leader_uuid: "ce91f820098f48c089495f536a430451" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1175bc70fda4a3092a6a69cce16d257" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce91f820098f48c089495f536a430451" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 32781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f20fec5f1a084e20979923ec0837c95c" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 40781 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:20.134660 22785 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 08:23:20.138514 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f20fec5f1a084e20979923ec0837c95c to finish bootstrapping
I20250626 08:23:20.151930 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ce91f820098f48c089495f536a430451 to finish bootstrapping
I20250626 08:23:20.165927 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f1175bc70fda4a3092a6a69cce16d257 to finish bootstrapping
I20250626 08:23:20.440991 25386 consensus_queue.cc:1035] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1175bc70fda4a3092a6a69cce16d257" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:23:20.464004 25399 consensus_queue.cc:1035] T d031cf4021e84b62a936b84505e12c23 P ce91f820098f48c089495f536a430451 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20fec5f1a084e20979923ec0837c95c" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 40781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 08:23:21.899386 24899 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:53464
I20250626 08:23:22.935406 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 24958
I20250626 08:23:22.965015 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 25091
I20250626 08:23:22.994488 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 25231
I20250626 08:23:23.022538 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 24866
2025-06-26T08:23:23Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (11160 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250626 08:23:23.077431 22785 test_util.cc:276] Using random seed: 1180484377
I20250626 08:23:23.081717 22785 ts_itest-base.cc:115] Starting cluster with:
I20250626 08:23:23.081898 22785 ts_itest-base.cc:116] --------------
I20250626 08:23:23.082064 22785 ts_itest-base.cc:117] 3 tablet servers
I20250626 08:23:23.082208 22785 ts_itest-base.cc:118] 3 replicas per TS
I20250626 08:23:23.082394 22785 ts_itest-base.cc:119] --------------
2025-06-26T08:23:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T08:23:23Z Disabled control of system clock
I20250626 08:23:23.124081 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:40933
--webserver_interface=127.22.64.126
--webserver_port=0
--builtin_ntp_servers=127.22.64.84:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.64.126:40933 with env {}
W20250626 08:23:23.436884 25435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:23.437476 25435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:23.437922 25435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:23.470618 25435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 08:23:23.470932 25435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:23.471139 25435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 08:23:23.471341 25435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 08:23:23.508152 25435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.64.126:40933
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:40933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.64.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:23.509614 25435 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:23.511298 25435 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:23.527882 25442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:23.527949 25441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:23.528753 25444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:24.716061 25435 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250626 08:23:24.716533 25443 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:23:24.721374 25435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:24.724841 25435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:24.726375 25435 hybrid_clock.cc:648] HybridClock initialized: now 1750926204726323 us; error 99 us; skew 500 ppm
I20250626 08:23:24.727241 25435 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:24.738690 25435 webserver.cc:469] Webserver started at http://127.22.64.126:36075/ using document root <none> and password file <none>
I20250626 08:23:24.739683 25435 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:24.739895 25435 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:24.740337 25435 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:23:24.744995 25435 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "59fe2f40639743bdaf06dae786592327"
format_stamp: "Formatted at 2025-06-26 08:23:24 on dist-test-slave-z1j3"
I20250626 08:23:24.746157 25435 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "59fe2f40639743bdaf06dae786592327"
format_stamp: "Formatted at 2025-06-26 08:23:24 on dist-test-slave-z1j3"
I20250626 08:23:24.753824 25435 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250626 08:23:24.759919 25451 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:24.761044 25435 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250626 08:23:24.761376 25435 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
uuid: "59fe2f40639743bdaf06dae786592327"
format_stamp: "Formatted at 2025-06-26 08:23:24 on dist-test-slave-z1j3"
I20250626 08:23:24.761781 25435 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:24.813545 25435 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:24.815292 25435 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:24.815775 25435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:24.888659 25435 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.126:40933
I20250626 08:23:24.888710 25502 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.126:40933 every 8 connection(s)
I20250626 08:23:24.891520 25435 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 08:23:24.896819 25503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:24.902247 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 25435
I20250626 08:23:24.902798 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 08:23:24.922371 25503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Bootstrap starting.
I20250626 08:23:24.929018 25503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Neither blocks nor log segments found. Creating new log.
I20250626 08:23:24.930920 25503 log.cc:826] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:24.936204 25503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: No bootstrap required, opened a new log
I20250626 08:23:24.953929 25503 raft_consensus.cc:357] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:24.954722 25503 raft_consensus.cc:383] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:23:24.954972 25503 raft_consensus.cc:738] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fe2f40639743bdaf06dae786592327, State: Initialized, Role: FOLLOWER
I20250626 08:23:24.955755 25503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:24.956281 25503 raft_consensus.cc:397] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:23:24.956562 25503 raft_consensus.cc:491] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:23:24.956863 25503 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:23:24.961102 25503 raft_consensus.cc:513] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:24.961855 25503 leader_election.cc:304] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59fe2f40639743bdaf06dae786592327; no voters:
I20250626 08:23:24.963739 25503 leader_election.cc:290] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:23:24.964563 25508 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:23:24.967108 25508 raft_consensus.cc:695] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 1 LEADER]: Becoming Leader. State: Replica: 59fe2f40639743bdaf06dae786592327, State: Running, Role: LEADER
I20250626 08:23:24.967923 25508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:24.969141 25503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:23:24.979174 25510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59fe2f40639743bdaf06dae786592327. Latest consensus state: current_term: 1 leader_uuid: "59fe2f40639743bdaf06dae786592327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } } }
I20250626 08:23:24.980422 25510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: This master's current role is: LEADER
I20250626 08:23:24.980474 25509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59fe2f40639743bdaf06dae786592327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } } }
I20250626 08:23:24.981616 25509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: This master's current role is: LEADER
I20250626 08:23:24.983901 25516 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:23:24.996901 25516 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:23:25.015571 25516 catalog_manager.cc:1349] Generated new cluster ID: 7f9f7a6de1c14fc99e2cba320babdec2
I20250626 08:23:25.015966 25516 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:23:25.030382 25516 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 08:23:25.031952 25516 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:23:25.043681 25516 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Generated new TSK 0
I20250626 08:23:25.044726 25516 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 08:23:25.066833 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.65:0
--local_ip_for_outbound_sockets=127.22.64.65
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:40933
--builtin_ntp_servers=127.22.64.84:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 08:23:25.393599 25527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:25.394142 25527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:25.394744 25527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:25.427966 25527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:25.428864 25527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.65
I20250626 08:23:25.464393 25527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:40933
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.65
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:25.465812 25527 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:25.467552 25527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:25.486711 25534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:25.487198 25533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:25.490073 25536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:26.694846 25535 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:23:26.694967 25527 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:23:26.696249 25527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:26.699247 25527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:26.700748 25527 hybrid_clock.cc:648] HybridClock initialized: now 1750926206700692 us; error 73 us; skew 500 ppm
I20250626 08:23:26.701613 25527 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:26.708564 25527 webserver.cc:469] Webserver started at http://127.22.64.65:36379/ using document root <none> and password file <none>
I20250626 08:23:26.709548 25527 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:26.709791 25527 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:26.710278 25527 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:23:26.714938 25527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fc5c7a6170284343a573042e7224d5e8"
format_stamp: "Formatted at 2025-06-26 08:23:26 on dist-test-slave-z1j3"
I20250626 08:23:26.716077 25527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fc5c7a6170284343a573042e7224d5e8"
format_stamp: "Formatted at 2025-06-26 08:23:26 on dist-test-slave-z1j3"
I20250626 08:23:26.723960 25527 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250626 08:23:26.730377 25543 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:26.731678 25527 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 08:23:26.732046 25527 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fc5c7a6170284343a573042e7224d5e8"
format_stamp: "Formatted at 2025-06-26 08:23:26 on dist-test-slave-z1j3"
I20250626 08:23:26.732401 25527 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:26.797263 25527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:26.798859 25527 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:26.799350 25527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:26.801960 25527 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:23:26.806160 25527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:23:26.806388 25527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250626 08:23:26.806619 25527 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:23:26.806756 25527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:26.945784 25527 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.65:37235
I20250626 08:23:26.945883 25655 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.65:37235 every 8 connection(s)
I20250626 08:23:26.948402 25527 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 08:23:26.954706 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 25527
I20250626 08:23:26.955219 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 08:23:26.961112 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.66:0
--local_ip_for_outbound_sockets=127.22.64.66
--webserver_interface=127.22.64.66
--webserver_port=0
--tserver_master_addrs=127.22.64.126:40933
--builtin_ntp_servers=127.22.64.84:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:23:26.969527 25656 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40933
I20250626 08:23:26.970111 25656 heartbeater.cc:461] Registering TS with master...
I20250626 08:23:26.971521 25656 heartbeater.cc:507] Master 127.22.64.126:40933 requested a full tablet report, sending...
I20250626 08:23:26.974571 25468 ts_manager.cc:194] Registered new tserver with Master: fc5c7a6170284343a573042e7224d5e8 (127.22.64.65:37235)
I20250626 08:23:26.976533 25468 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.65:37223
W20250626 08:23:27.269838 25660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:27.270395 25660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:27.270982 25660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:27.303278 25660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:27.304160 25660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.66
I20250626 08:23:27.340432 25660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.64.66
--webserver_port=0
--tserver_master_addrs=127.22.64.126:40933
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.66
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:27.342134 25660 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:27.343931 25660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:27.360101 25667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:27.980329 25656 heartbeater.cc:499] Master 127.22.64.126:40933 was elected leader, sending a full tablet report...
W20250626 08:23:27.360108 25666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:27.361173 25669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:28.533250 25668 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:23:28.533303 25660 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:23:28.538208 25660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:28.540577 25660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:28.541953 25660 hybrid_clock.cc:648] HybridClock initialized: now 1750926208541910 us; error 63 us; skew 500 ppm
I20250626 08:23:28.542866 25660 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:28.549664 25660 webserver.cc:469] Webserver started at http://127.22.64.66:46315/ using document root <none> and password file <none>
I20250626 08:23:28.550864 25660 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:28.551080 25660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:28.551506 25660 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:23:28.556087 25660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "530bcc5617084355ac56c210c23445fc"
format_stamp: "Formatted at 2025-06-26 08:23:28 on dist-test-slave-z1j3"
I20250626 08:23:28.557247 25660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "530bcc5617084355ac56c210c23445fc"
format_stamp: "Formatted at 2025-06-26 08:23:28 on dist-test-slave-z1j3"
I20250626 08:23:28.565099 25660 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250626 08:23:28.571296 25677 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:28.572440 25660 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250626 08:23:28.572813 25660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "530bcc5617084355ac56c210c23445fc"
format_stamp: "Formatted at 2025-06-26 08:23:28 on dist-test-slave-z1j3"
I20250626 08:23:28.573143 25660 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:28.633136 25660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:28.634722 25660 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:28.635176 25660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:28.637799 25660 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:23:28.642156 25660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:23:28.642443 25660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250626 08:23:28.642738 25660 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:23:28.642905 25660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:28.781131 25660 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.66:43107
I20250626 08:23:28.781245 25789 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.66:43107 every 8 connection(s)
I20250626 08:23:28.783794 25660 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 08:23:28.792827 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 25660
I20250626 08:23:28.793418 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 08:23:28.800209 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.67:0
--local_ip_for_outbound_sockets=127.22.64.67
--webserver_interface=127.22.64.67
--webserver_port=0
--tserver_master_addrs=127.22.64.126:40933
--builtin_ntp_servers=127.22.64.84:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:23:28.805732 25790 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40933
I20250626 08:23:28.806345 25790 heartbeater.cc:461] Registering TS with master...
I20250626 08:23:28.807801 25790 heartbeater.cc:507] Master 127.22.64.126:40933 requested a full tablet report, sending...
I20250626 08:23:28.810492 25468 ts_manager.cc:194] Registered new tserver with Master: 530bcc5617084355ac56c210c23445fc (127.22.64.66:43107)
I20250626 08:23:28.811784 25468 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.66:58265
W20250626 08:23:29.109309 25794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:29.109827 25794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:29.110412 25794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:29.142740 25794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:29.143628 25794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.67
I20250626 08:23:29.182001 25794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.64.67
--webserver_port=0
--tserver_master_addrs=127.22.64.126:40933
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.67
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:29.183460 25794 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:29.185158 25794 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:29.201273 25801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:29.815428 25790 heartbeater.cc:499] Master 127.22.64.126:40933 was elected leader, sending a full tablet report...
W20250626 08:23:29.202991 25803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:29.201514 25800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:30.403801 25802 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1200 milliseconds
I20250626 08:23:30.403970 25794 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:23:30.405277 25794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:30.407640 25794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:30.409053 25794 hybrid_clock.cc:648] HybridClock initialized: now 1750926210409019 us; error 40 us; skew 500 ppm
I20250626 08:23:30.409987 25794 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:30.421496 25794 webserver.cc:469] Webserver started at http://127.22.64.67:40387/ using document root <none> and password file <none>
I20250626 08:23:30.422577 25794 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:30.422823 25794 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:30.423302 25794 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:23:30.427866 25794 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a47f6e68e2e3439f895bc95cdec80597"
format_stamp: "Formatted at 2025-06-26 08:23:30 on dist-test-slave-z1j3"
I20250626 08:23:30.429001 25794 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a47f6e68e2e3439f895bc95cdec80597"
format_stamp: "Formatted at 2025-06-26 08:23:30 on dist-test-slave-z1j3"
I20250626 08:23:30.436295 25794 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20250626 08:23:30.442538 25810 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:30.443768 25794 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250626 08:23:30.444135 25794 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a47f6e68e2e3439f895bc95cdec80597"
format_stamp: "Formatted at 2025-06-26 08:23:30 on dist-test-slave-z1j3"
I20250626 08:23:30.444482 25794 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:30.490176 25794 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:30.491739 25794 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:30.492184 25794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:30.494727 25794 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:23:30.498921 25794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:23:30.499157 25794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:30.499426 25794 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:23:30.499588 25794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:30.642302 25794 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.67:34135
I20250626 08:23:30.642483 25922 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.67:34135 every 8 connection(s)
I20250626 08:23:30.644901 25794 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 08:23:30.650274 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 25794
I20250626 08:23:30.650776 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 08:23:30.676028 25923 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40933
I20250626 08:23:30.676474 25923 heartbeater.cc:461] Registering TS with master...
I20250626 08:23:30.677588 25923 heartbeater.cc:507] Master 127.22.64.126:40933 requested a full tablet report, sending...
I20250626 08:23:30.680061 25468 ts_manager.cc:194] Registered new tserver with Master: a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135)
I20250626 08:23:30.681344 25468 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.67:45763
I20250626 08:23:30.689486 22785 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 08:23:30.734483 22785 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 08:23:30.734839 22785 test_util.cc:276] Using random seed: 1188141795
I20250626 08:23:30.778901 25468 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34708:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 08:23:30.824627 25858 tablet_service.cc:1468] Processing CreateTablet for tablet bb71af2b445a4783958bf42caa5b7d6b (DEFAULT_TABLE table=TestTable [id=23e1cbaca8e44ead9b7fd7a044cad489]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:23:30.826143 25858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb71af2b445a4783958bf42caa5b7d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:30.846195 25943 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Bootstrap starting.
I20250626 08:23:30.851881 25943 tablet_bootstrap.cc:654] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Neither blocks nor log segments found. Creating new log.
I20250626 08:23:30.853792 25943 log.cc:826] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:30.858814 25943 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: No bootstrap required, opened a new log
I20250626 08:23:30.859297 25943 ts_tablet_manager.cc:1397] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20250626 08:23:30.877337 25943 raft_consensus.cc:357] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } }
I20250626 08:23:30.878053 25943 raft_consensus.cc:383] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:23:30.878373 25943 raft_consensus.cc:738] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47f6e68e2e3439f895bc95cdec80597, State: Initialized, Role: FOLLOWER
I20250626 08:23:30.879325 25943 consensus_queue.cc:260] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } }
I20250626 08:23:30.879837 25943 raft_consensus.cc:397] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:23:30.880069 25943 raft_consensus.cc:491] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:23:30.880323 25943 raft_consensus.cc:3058] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:23:30.885344 25943 raft_consensus.cc:513] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } }
I20250626 08:23:30.886114 25943 leader_election.cc:304] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a47f6e68e2e3439f895bc95cdec80597; no voters:
I20250626 08:23:30.888209 25943 leader_election.cc:290] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:23:30.888684 25945 raft_consensus.cc:2802] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:23:30.892109 25923 heartbeater.cc:499] Master 127.22.64.126:40933 was elected leader, sending a full tablet report...
I20250626 08:23:30.892575 25945 raft_consensus.cc:695] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 1 LEADER]: Becoming Leader. State: Replica: a47f6e68e2e3439f895bc95cdec80597, State: Running, Role: LEADER
I20250626 08:23:30.893631 25945 consensus_queue.cc:237] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } }
I20250626 08:23:30.894112 25943 ts_tablet_manager.cc:1428] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Time spent starting tablet: real 0.035s user 0.020s sys 0.014s
W20250626 08:23:30.899890 25924 tablet.cc:2378] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:23:30.908396 25467 catalog_manager.cc:5582] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67). New cstate: current_term: 1 leader_uuid: "a47f6e68e2e3439f895bc95cdec80597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } health_report { overall_health: HEALTHY } } }
I20250626 08:23:31.074900 22785 test_util.cc:276] Using random seed: 1188481834
I20250626 08:23:31.097693 25464 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34710:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 08:23:31.129246 25591 tablet_service.cc:1468] Processing CreateTablet for tablet ae1749c72e1c46f3b26e946563289089 (DEFAULT_TABLE table=TestTable1 [id=4153cccf6a314fcab04bdb28756cf715]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:23:31.130964 25591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1749c72e1c46f3b26e946563289089. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:31.151566 25964 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Bootstrap starting.
I20250626 08:23:31.158255 25964 tablet_bootstrap.cc:654] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Neither blocks nor log segments found. Creating new log.
I20250626 08:23:31.160218 25964 log.cc:826] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:31.165329 25964 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: No bootstrap required, opened a new log
I20250626 08:23:31.165876 25964 ts_tablet_manager.cc:1397] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20250626 08:23:31.185958 25964 raft_consensus.cc:357] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } }
I20250626 08:23:31.186714 25964 raft_consensus.cc:383] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:23:31.186934 25964 raft_consensus.cc:738] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5c7a6170284343a573042e7224d5e8, State: Initialized, Role: FOLLOWER
I20250626 08:23:31.187666 25964 consensus_queue.cc:260] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } }
I20250626 08:23:31.188282 25964 raft_consensus.cc:397] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:23:31.188535 25964 raft_consensus.cc:491] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:23:31.188804 25964 raft_consensus.cc:3058] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:23:31.193603 25964 raft_consensus.cc:513] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } }
I20250626 08:23:31.194381 25964 leader_election.cc:304] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc5c7a6170284343a573042e7224d5e8; no voters:
I20250626 08:23:31.196154 25964 leader_election.cc:290] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:23:31.196593 25966 raft_consensus.cc:2802] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:23:31.199107 25966 raft_consensus.cc:695] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 1 LEADER]: Becoming Leader. State: Replica: fc5c7a6170284343a573042e7224d5e8, State: Running, Role: LEADER
I20250626 08:23:31.200414 25964 ts_tablet_manager.cc:1428] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Time spent starting tablet: real 0.034s user 0.032s sys 0.004s
I20250626 08:23:31.200127 25966 consensus_queue.cc:237] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } }
I20250626 08:23:31.212282 25464 catalog_manager.cc:5582] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc5c7a6170284343a573042e7224d5e8 (127.22.64.65). New cstate: current_term: 1 leader_uuid: "fc5c7a6170284343a573042e7224d5e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } health_report { overall_health: HEALTHY } } }
I20250626 08:23:31.419943 22785 test_util.cc:276] Using random seed: 1188826884
I20250626 08:23:31.451900 25462 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34722:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 08:23:31.484357 25725 tablet_service.cc:1468] Processing CreateTablet for tablet 9761923646a248148bee793cbae0b5ec (DEFAULT_TABLE table=TestTable2 [id=5213ff601e694deda4b7cefeaee2ffb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:23:31.485869 25725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9761923646a248148bee793cbae0b5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:31.507014 25985 tablet_bootstrap.cc:492] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Bootstrap starting.
I20250626 08:23:31.513311 25985 tablet_bootstrap.cc:654] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Neither blocks nor log segments found. Creating new log.
I20250626 08:23:31.515304 25985 log.cc:826] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:31.520249 25985 tablet_bootstrap.cc:492] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: No bootstrap required, opened a new log
I20250626 08:23:31.520849 25985 ts_tablet_manager.cc:1397] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20250626 08:23:31.538978 25985 raft_consensus.cc:357] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } }
I20250626 08:23:31.539631 25985 raft_consensus.cc:383] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:23:31.539885 25985 raft_consensus.cc:738] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530bcc5617084355ac56c210c23445fc, State: Initialized, Role: FOLLOWER
I20250626 08:23:31.540640 25985 consensus_queue.cc:260] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } }
I20250626 08:23:31.541191 25985 raft_consensus.cc:397] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:23:31.541482 25985 raft_consensus.cc:491] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:23:31.541788 25985 raft_consensus.cc:3058] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:23:31.546307 25985 raft_consensus.cc:513] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } }
I20250626 08:23:31.547125 25985 leader_election.cc:304] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 530bcc5617084355ac56c210c23445fc; no voters:
I20250626 08:23:31.549177 25985 leader_election.cc:290] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:23:31.549676 25987 raft_consensus.cc:2802] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:23:31.552060 25987 raft_consensus.cc:695] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 1 LEADER]: Becoming Leader. State: Replica: 530bcc5617084355ac56c210c23445fc, State: Running, Role: LEADER
I20250626 08:23:31.553319 25985 ts_tablet_manager.cc:1428] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Time spent starting tablet: real 0.032s user 0.027s sys 0.006s
I20250626 08:23:31.553568 25987 consensus_queue.cc:237] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } }
I20250626 08:23:31.565229 25462 catalog_manager.cc:5582] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 530bcc5617084355ac56c210c23445fc (127.22.64.66). New cstate: current_term: 1 leader_uuid: "530bcc5617084355ac56c210c23445fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } health_report { overall_health: HEALTHY } } }
I20250626 08:23:31.817731 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 25435
W20250626 08:23:31.927857 25923 heartbeater.cc:646] Failed to heartbeat to 127.22.64.126:40933 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.64.126:40933: connect: Connection refused (error 111)
W20250626 08:23:32.228094 25656 heartbeater.cc:646] Failed to heartbeat to 127.22.64.126:40933 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.64.126:40933: connect: Connection refused (error 111)
W20250626 08:23:32.585484 25790 heartbeater.cc:646] Failed to heartbeat to 127.22.64.126:40933 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.64.126:40933: connect: Connection refused (error 111)
I20250626 08:23:36.739629 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 25527
I20250626 08:23:36.764381 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 25660
I20250626 08:23:36.792865 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 25794
I20250626 08:23:36.822481 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:40933
--webserver_interface=127.22.64.126
--webserver_port=36075
--builtin_ntp_servers=127.22.64.84:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.64.126:40933 with env {}
W20250626 08:23:37.126683 26064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:37.127323 26064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:37.127921 26064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:37.159071 26064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 08:23:37.159437 26064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:37.159705 26064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 08:23:37.159965 26064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 08:23:37.195952 26064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.64.126:40933
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:40933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.64.126
--webserver_port=36075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:37.197373 26064 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:37.199193 26064 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:37.216374 26073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:37.216801 26071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:37.218803 26064 server_base.cc:1048] running on GCE node
W20250626 08:23:37.218147 26070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:38.420368 26064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:38.423252 26064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:38.424645 26064 hybrid_clock.cc:648] HybridClock initialized: now 1750926218424601 us; error 47 us; skew 500 ppm
I20250626 08:23:38.425498 26064 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:38.436061 26064 webserver.cc:469] Webserver started at http://127.22.64.126:36075/ using document root <none> and password file <none>
I20250626 08:23:38.437117 26064 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:38.437371 26064 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:38.445878 26064 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250626 08:23:38.450779 26081 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:38.451993 26064 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250626 08:23:38.452327 26064 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
uuid: "59fe2f40639743bdaf06dae786592327"
format_stamp: "Formatted at 2025-06-26 08:23:24 on dist-test-slave-z1j3"
I20250626 08:23:38.454413 26064 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:38.516907 26064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:38.518517 26064 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:38.518949 26064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:38.592823 26064 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.126:40933
I20250626 08:23:38.592919 26132 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.126:40933 every 8 connection(s)
I20250626 08:23:38.595851 26064 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 08:23:38.596794 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 26064
I20250626 08:23:38.599365 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.65:37235
--local_ip_for_outbound_sockets=127.22.64.65
--tserver_master_addrs=127.22.64.126:40933
--webserver_port=36379
--webserver_interface=127.22.64.65
--builtin_ntp_servers=127.22.64.84:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:23:38.612252 26133 sys_catalog.cc:263] Verifying existing consensus state
I20250626 08:23:38.617247 26133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Bootstrap starting.
I20250626 08:23:38.628091 26133 log.cc:826] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:38.676587 26133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:38.677470 26133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Bootstrap complete.
I20250626 08:23:38.699167 26133 raft_consensus.cc:357] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:38.701462 26133 raft_consensus.cc:738] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fe2f40639743bdaf06dae786592327, State: Initialized, Role: FOLLOWER
I20250626 08:23:38.702446 26133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:38.703020 26133 raft_consensus.cc:397] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:23:38.703310 26133 raft_consensus.cc:491] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:23:38.703627 26133 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:38.709555 26133 raft_consensus.cc:513] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:38.710264 26133 leader_election.cc:304] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59fe2f40639743bdaf06dae786592327; no voters:
I20250626 08:23:38.712541 26133 leader_election.cc:290] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250626 08:23:38.713218 26137 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:38.716590 26137 raft_consensus.cc:695] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 3 LEADER]: Becoming Leader. State: Replica: 59fe2f40639743bdaf06dae786592327, State: Running, Role: LEADER
I20250626 08:23:38.717458 26133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:23:38.717566 26137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:38.731448 26138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "59fe2f40639743bdaf06dae786592327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } } }
I20250626 08:23:38.732575 26138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: This master's current role is: LEADER
I20250626 08:23:38.733430 26139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59fe2f40639743bdaf06dae786592327. Latest consensus state: current_term: 3 leader_uuid: "59fe2f40639743bdaf06dae786592327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } } }
I20250626 08:23:38.735736 26139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: This master's current role is: LEADER
I20250626 08:23:38.746798 26145 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:23:38.762384 26145 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=5213ff601e694deda4b7cefeaee2ffb9]
I20250626 08:23:38.765097 26145 catalog_manager.cc:671] Loaded metadata for table TestTable [id=88494ae7328c48d8871fb118b5311ca5]
I20250626 08:23:38.767091 26145 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=fd17003f89324fb3bf2e63608349bf57]
I20250626 08:23:38.777581 26145 tablet_loader.cc:96] loaded metadata for tablet 9761923646a248148bee793cbae0b5ec (table TestTable2 [id=5213ff601e694deda4b7cefeaee2ffb9])
I20250626 08:23:38.779843 26145 tablet_loader.cc:96] loaded metadata for tablet ae1749c72e1c46f3b26e946563289089 (table TestTable1 [id=fd17003f89324fb3bf2e63608349bf57])
I20250626 08:23:38.781658 26145 tablet_loader.cc:96] loaded metadata for tablet bb71af2b445a4783958bf42caa5b7d6b (table TestTable [id=88494ae7328c48d8871fb118b5311ca5])
I20250626 08:23:38.783608 26145 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:23:38.792922 26145 catalog_manager.cc:1261] Loaded cluster ID: 7f9f7a6de1c14fc99e2cba320babdec2
I20250626 08:23:38.793185 26145 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:23:38.804299 26145 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:23:38.811810 26145 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Loaded TSK: 0
I20250626 08:23:38.813565 26145 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250626 08:23:38.956298 26135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:38.956887 26135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:38.957427 26135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:38.989704 26135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:38.990765 26135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.65
I20250626 08:23:39.028133 26135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.65:37235
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.64.65
--webserver_port=36379
--tserver_master_addrs=127.22.64.126:40933
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.65
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:39.029541 26135 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:39.031366 26135 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:39.051190 26160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:39.052899 26163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:39.053452 26135 server_base.cc:1048] running on GCE node
W20250626 08:23:39.051851 26161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:40.252910 26135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:40.255839 26135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:40.257339 26135 hybrid_clock.cc:648] HybridClock initialized: now 1750926220257267 us; error 85 us; skew 500 ppm
I20250626 08:23:40.258265 26135 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:40.265475 26135 webserver.cc:469] Webserver started at http://127.22.64.65:36379/ using document root <none> and password file <none>
I20250626 08:23:40.266523 26135 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:40.266772 26135 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:40.275506 26135 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.003s
I20250626 08:23:40.280933 26170 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:40.282151 26135 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.002s
I20250626 08:23:40.282549 26135 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fc5c7a6170284343a573042e7224d5e8"
format_stamp: "Formatted at 2025-06-26 08:23:26 on dist-test-slave-z1j3"
I20250626 08:23:40.284611 26135 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:40.336143 26135 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:40.337671 26135 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:40.338122 26135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:40.341265 26135 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:23:40.347755 26177 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 08:23:40.356169 26135 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 08:23:40.356417 26135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.002s sys 0.000s
I20250626 08:23:40.356736 26135 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 08:23:40.361744 26135 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 08:23:40.362025 26135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250626 08:23:40.362464 26177 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Bootstrap starting.
I20250626 08:23:40.441510 26177 log.cc:826] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:40.572314 26135 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.65:37235
I20250626 08:23:40.572516 26284 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.65:37235 every 8 connection(s)
I20250626 08:23:40.573175 26177 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:40.573935 26177 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Bootstrap complete.
I20250626 08:23:40.575278 26177 ts_tablet_manager.cc:1397] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Time spent bootstrapping tablet: real 0.213s user 0.172s sys 0.037s
I20250626 08:23:40.575953 26135 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 08:23:40.578043 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 26135
I20250626 08:23:40.579953 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.66:43107
--local_ip_for_outbound_sockets=127.22.64.66
--tserver_master_addrs=127.22.64.126:40933
--webserver_port=46315
--webserver_interface=127.22.64.66
--builtin_ntp_servers=127.22.64.84:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:23:40.623281 26177 raft_consensus.cc:357] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } }
I20250626 08:23:40.626346 26177 raft_consensus.cc:738] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5c7a6170284343a573042e7224d5e8, State: Initialized, Role: FOLLOWER
I20250626 08:23:40.627557 26177 consensus_queue.cc:260] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } }
I20250626 08:23:40.628304 26177 raft_consensus.cc:397] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:23:40.628690 26177 raft_consensus.cc:491] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:23:40.629213 26177 raft_consensus.cc:3058] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:40.637152 26177 raft_consensus.cc:513] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } }
I20250626 08:23:40.638127 26177 leader_election.cc:304] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc5c7a6170284343a573042e7224d5e8; no voters:
I20250626 08:23:40.645327 26177 leader_election.cc:290] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 08:23:40.646284 26285 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40933
I20250626 08:23:40.646863 26285 heartbeater.cc:461] Registering TS with master...
I20250626 08:23:40.648077 26285 heartbeater.cc:507] Master 127.22.64.126:40933 requested a full tablet report, sending...
I20250626 08:23:40.648926 26290 raft_consensus.cc:2802] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:40.655951 26177 ts_tablet_manager.cc:1428] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Time spent starting tablet: real 0.080s user 0.045s sys 0.008s
I20250626 08:23:40.657159 26290 raft_consensus.cc:695] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 LEADER]: Becoming Leader. State: Replica: fc5c7a6170284343a573042e7224d5e8, State: Running, Role: LEADER
I20250626 08:23:40.658023 26290 consensus_queue.cc:237] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } }
I20250626 08:23:40.671465 26098 ts_manager.cc:194] Registered new tserver with Master: fc5c7a6170284343a573042e7224d5e8 (127.22.64.65:37235)
I20250626 08:23:40.677659 26098 catalog_manager.cc:5582] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 reported cstate change: term changed from 0 to 2, leader changed from <none> to fc5c7a6170284343a573042e7224d5e8 (127.22.64.65), VOTER fc5c7a6170284343a573042e7224d5e8 (127.22.64.65) added. New cstate: current_term: 2 leader_uuid: "fc5c7a6170284343a573042e7224d5e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } health_report { overall_health: HEALTHY } } }
W20250626 08:23:40.729125 26098 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ae1749c72e1c46f3b26e946563289089 with cas_config_opid_index -1: no extra replica candidate found for tablet ae1749c72e1c46f3b26e946563289089 (table TestTable1 [id=fd17003f89324fb3bf2e63608349bf57]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250626 08:23:40.731511 26098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.65:36945
I20250626 08:23:40.735558 26285 heartbeater.cc:499] Master 127.22.64.126:40933 was elected leader, sending a full tablet report...
W20250626 08:23:40.928018 26287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:40.928567 26287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:40.929072 26287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:40.961408 26287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:40.962345 26287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.66
I20250626 08:23:40.998155 26287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.66:43107
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.64.66
--webserver_port=46315
--tserver_master_addrs=127.22.64.126:40933
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.66
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:40.999610 26287 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:41.001260 26287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:41.016563 26304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:41.016628 26305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:41.019703 26307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:42.200359 26306 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:23:42.200419 26287 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:23:42.204233 26287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:42.207078 26287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:42.208559 26287 hybrid_clock.cc:648] HybridClock initialized: now 1750926222208513 us; error 66 us; skew 500 ppm
I20250626 08:23:42.209400 26287 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:42.216879 26287 webserver.cc:469] Webserver started at http://127.22.64.66:46315/ using document root <none> and password file <none>
I20250626 08:23:42.217806 26287 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:42.218025 26287 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:42.227442 26287 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20250626 08:23:42.232743 26314 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:42.233915 26287 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250626 08:23:42.234278 26287 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "530bcc5617084355ac56c210c23445fc"
format_stamp: "Formatted at 2025-06-26 08:23:28 on dist-test-slave-z1j3"
I20250626 08:23:42.236280 26287 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:42.289059 26287 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:42.290627 26287 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:42.291102 26287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:42.293634 26287 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:23:42.299304 26321 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 08:23:42.306806 26287 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 08:23:42.307053 26287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250626 08:23:42.307313 26287 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 08:23:42.312088 26287 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 08:23:42.312288 26287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.002s
I20250626 08:23:42.312644 26321 tablet_bootstrap.cc:492] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Bootstrap starting.
I20250626 08:23:42.366832 26321 log.cc:826] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:42.474869 26321 tablet_bootstrap.cc:492] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:42.475725 26321 tablet_bootstrap.cc:492] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Bootstrap complete.
I20250626 08:23:42.477487 26321 ts_tablet_manager.cc:1397] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Time spent bootstrapping tablet: real 0.165s user 0.132s sys 0.029s
I20250626 08:23:42.494844 26321 raft_consensus.cc:357] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } }
I20250626 08:23:42.497025 26321 raft_consensus.cc:738] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530bcc5617084355ac56c210c23445fc, State: Initialized, Role: FOLLOWER
I20250626 08:23:42.497787 26287 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.66:43107
I20250626 08:23:42.497965 26428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.66:43107 every 8 connection(s)
I20250626 08:23:42.497895 26321 consensus_queue.cc:260] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } }
I20250626 08:23:42.498593 26321 raft_consensus.cc:397] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:23:42.498857 26321 raft_consensus.cc:491] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:23:42.499146 26321 raft_consensus.cc:3058] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:42.500586 26287 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 08:23:42.507073 26321 raft_consensus.cc:513] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } }
I20250626 08:23:42.507853 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 26287
I20250626 08:23:42.508271 26321 leader_election.cc:304] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 530bcc5617084355ac56c210c23445fc; no voters:
I20250626 08:23:42.509788 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.67:34135
--local_ip_for_outbound_sockets=127.22.64.67
--tserver_master_addrs=127.22.64.126:40933
--webserver_port=40387
--webserver_interface=127.22.64.67
--builtin_ntp_servers=127.22.64.84:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:23:42.510614 26321 leader_election.cc:290] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 08:23:42.510937 26433 raft_consensus.cc:2802] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:42.519035 26433 raft_consensus.cc:695] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 2 LEADER]: Becoming Leader. State: Replica: 530bcc5617084355ac56c210c23445fc, State: Running, Role: LEADER
I20250626 08:23:42.520334 26321 ts_tablet_manager.cc:1428] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Time spent starting tablet: real 0.043s user 0.034s sys 0.009s
I20250626 08:23:42.520288 26433 consensus_queue.cc:237] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } }
I20250626 08:23:42.531245 26429 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40933
I20250626 08:23:42.531797 26429 heartbeater.cc:461] Registering TS with master...
I20250626 08:23:42.533249 26429 heartbeater.cc:507] Master 127.22.64.126:40933 requested a full tablet report, sending...
I20250626 08:23:42.536818 26098 ts_manager.cc:194] Registered new tserver with Master: 530bcc5617084355ac56c210c23445fc (127.22.64.66:43107)
I20250626 08:23:42.537845 26098 catalog_manager.cc:5582] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "530bcc5617084355ac56c210c23445fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } health_report { overall_health: HEALTHY } } }
I20250626 08:23:42.548811 26098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.66:38931
I20250626 08:23:42.552918 26429 heartbeater.cc:499] Master 127.22.64.126:40933 was elected leader, sending a full tablet report...
W20250626 08:23:42.828576 26436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:42.829067 26436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:42.829530 26436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:42.861272 26436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:42.862332 26436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.67
I20250626 08:23:42.897981 26436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.67:34135
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.64.67
--webserver_port=40387
--tserver_master_addrs=127.22.64.126:40933
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.67
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:42.899359 26436 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:42.901090 26436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:42.917027 26448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:43.045257 26240 consensus_queue.cc:237] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: NON_VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: true } }
I20250626 08:23:43.052654 26458 raft_consensus.cc:2953] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER 530bcc5617084355ac56c210c23445fc (127.22.64.66) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: NON_VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: true } } }
I20250626 08:23:43.072481 26084 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ae1749c72e1c46f3b26e946563289089 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250626 08:23:43.077538 26098 catalog_manager.cc:5582] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 reported cstate change: config changed from index -1 to 9, NON_VOTER 530bcc5617084355ac56c210c23445fc (127.22.64.66) added. New cstate: current_term: 2 leader_uuid: "fc5c7a6170284343a573042e7224d5e8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: NON_VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 08:23:43.081879 26172 consensus_peers.cc:487] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 -> Peer 530bcc5617084355ac56c210c23445fc (127.22.64.66:43107): Couldn't send request to peer 530bcc5617084355ac56c210c23445fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae1749c72e1c46f3b26e946563289089. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:43.523439 26465 ts_tablet_manager.cc:927] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: Initiating tablet copy from peer fc5c7a6170284343a573042e7224d5e8 (127.22.64.65:37235)
I20250626 08:23:43.526306 26465 tablet_copy_client.cc:323] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: tablet copy: Beginning tablet copy session from remote peer at address 127.22.64.65:37235
I20250626 08:23:43.543710 26260 tablet_copy_service.cc:140] P fc5c7a6170284343a573042e7224d5e8: Received BeginTabletCopySession request for tablet ae1749c72e1c46f3b26e946563289089 from peer 530bcc5617084355ac56c210c23445fc ({username='slave'} at 127.22.64.66:42899)
I20250626 08:23:43.544656 26260 tablet_copy_service.cc:161] P fc5c7a6170284343a573042e7224d5e8: Beginning new tablet copy session on tablet ae1749c72e1c46f3b26e946563289089 from peer 530bcc5617084355ac56c210c23445fc at {username='slave'} at 127.22.64.66:42899: session id = 530bcc5617084355ac56c210c23445fc-ae1749c72e1c46f3b26e946563289089
I20250626 08:23:43.556277 26260 tablet_copy_source_session.cc:215] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:43.561909 26465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1749c72e1c46f3b26e946563289089. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:43.573117 26465 tablet_copy_client.cc:806] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:43.573848 26465 tablet_copy_client.cc:670] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:43.577664 26465 tablet_copy_client.cc:538] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:43.583899 26465 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: Bootstrap starting.
I20250626 08:23:43.694165 26465 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:43.694983 26465 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: Bootstrap complete.
I20250626 08:23:43.695451 26465 ts_tablet_manager.cc:1397] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: Time spent bootstrapping tablet: real 0.112s user 0.085s sys 0.025s
I20250626 08:23:43.697121 26465 raft_consensus.cc:357] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: NON_VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: true } }
I20250626 08:23:43.697551 26465 raft_consensus.cc:738] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 530bcc5617084355ac56c210c23445fc, State: Initialized, Role: LEARNER
I20250626 08:23:43.697966 26465 consensus_queue.cc:260] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: NON_VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: true } }
I20250626 08:23:43.700865 26465 ts_tablet_manager.cc:1428] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250626 08:23:43.702378 26260 tablet_copy_service.cc:342] P fc5c7a6170284343a573042e7224d5e8: Request end of tablet copy session 530bcc5617084355ac56c210c23445fc-ae1749c72e1c46f3b26e946563289089 received from {username='slave'} at 127.22.64.66:42899
I20250626 08:23:43.702826 26260 tablet_copy_service.cc:434] P fc5c7a6170284343a573042e7224d5e8: ending tablet copy session 530bcc5617084355ac56c210c23445fc-ae1749c72e1c46f3b26e946563289089 on tablet ae1749c72e1c46f3b26e946563289089 with peer 530bcc5617084355ac56c210c23445fc
I20250626 08:23:43.923447 26384 raft_consensus.cc:1215] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20250626 08:23:42.917418 26449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:42.918896 26451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:44.136186 26450 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:23:44.136237 26436 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:23:44.140573 26436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:44.143481 26436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:44.144929 26436 hybrid_clock.cc:648] HybridClock initialized: now 1750926224144893 us; error 62 us; skew 500 ppm
I20250626 08:23:44.145815 26436 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:44.158052 26436 webserver.cc:469] Webserver started at http://127.22.64.67:40387/ using document root <none> and password file <none>
I20250626 08:23:44.159103 26436 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:44.159354 26436 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:44.167515 26436 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250626 08:23:44.172798 26476 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:44.173903 26436 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250626 08:23:44.174293 26436 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a47f6e68e2e3439f895bc95cdec80597"
format_stamp: "Formatted at 2025-06-26 08:23:30 on dist-test-slave-z1j3"
I20250626 08:23:44.176373 26436 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:44.238603 26436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:44.240141 26436 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:44.240617 26436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:44.243233 26436 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:23:44.249612 26483 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 08:23:44.257318 26436 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 08:23:44.257582 26436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.001s
I20250626 08:23:44.257858 26436 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 08:23:44.262650 26436 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 08:23:44.262862 26436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250626 08:23:44.263237 26483 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Bootstrap starting.
I20250626 08:23:44.320161 26483 log.cc:826] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:44.371206 26458 raft_consensus.cc:1062] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: attempting to promote NON_VOTER 530bcc5617084355ac56c210c23445fc to VOTER
I20250626 08:23:44.373106 26458 consensus_queue.cc:237] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:23:44.380745 26384 raft_consensus.cc:1273] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 2 LEARNER]: Refusing update from remote peer fc5c7a6170284343a573042e7224d5e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250626 08:23:44.382611 26458 consensus_queue.cc:1035] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250626 08:23:44.393558 26458 raft_consensus.cc:2953] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 530bcc5617084355ac56c210c23445fc (127.22.64.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } }
I20250626 08:23:44.395989 26384 raft_consensus.cc:2953] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 530bcc5617084355ac56c210c23445fc (127.22.64.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } }
I20250626 08:23:44.410284 26097 catalog_manager.cc:5582] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc reported cstate change: config changed from index 9 to 10, 530bcc5617084355ac56c210c23445fc (127.22.64.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "fc5c7a6170284343a573042e7224d5e8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } }
I20250626 08:23:44.422060 26483 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:44.423252 26483 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Bootstrap complete.
I20250626 08:23:44.426384 26483 ts_tablet_manager.cc:1397] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Time spent bootstrapping tablet: real 0.164s user 0.119s sys 0.040s
I20250626 08:23:44.449445 26483 raft_consensus.cc:357] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } }
I20250626 08:23:44.452405 26483 raft_consensus.cc:738] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47f6e68e2e3439f895bc95cdec80597, State: Initialized, Role: FOLLOWER
I20250626 08:23:44.453394 26483 consensus_queue.cc:260] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } }
I20250626 08:23:44.453883 26483 raft_consensus.cc:397] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:23:44.454146 26483 raft_consensus.cc:491] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:23:44.454604 26483 raft_consensus.cc:3058] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:44.460755 26483 raft_consensus.cc:513] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } }
I20250626 08:23:44.461566 26483 leader_election.cc:304] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a47f6e68e2e3439f895bc95cdec80597; no voters:
I20250626 08:23:44.464419 26483 leader_election.cc:290] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 08:23:44.464920 26591 raft_consensus.cc:2802] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:44.468881 26591 raft_consensus.cc:695] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 2 LEADER]: Becoming Leader. State: Replica: a47f6e68e2e3439f895bc95cdec80597, State: Running, Role: LEADER
I20250626 08:23:44.469156 26483 ts_tablet_manager.cc:1428] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Time spent starting tablet: real 0.042s user 0.028s sys 0.011s
I20250626 08:23:44.470144 26591 consensus_queue.cc:237] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } }
I20250626 08:23:44.479513 26436 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.67:34135
I20250626 08:23:44.480101 26601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.67:34135 every 8 connection(s)
I20250626 08:23:44.482374 26436 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 08:23:44.483700 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 26436
I20250626 08:23:44.522704 26602 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40933
I20250626 08:23:44.523216 26602 heartbeater.cc:461] Registering TS with master...
I20250626 08:23:44.524580 26602 heartbeater.cc:507] Master 127.22.64.126:40933 requested a full tablet report, sending...
I20250626 08:23:44.528384 26097 ts_manager.cc:194] Registered new tserver with Master: a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135)
I20250626 08:23:44.529343 26097 catalog_manager.cc:5582] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 reported cstate change: term changed from 0 to 2, leader changed from <none> to a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67), VOTER a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67) added. New cstate: current_term: 2 leader_uuid: "a47f6e68e2e3439f895bc95cdec80597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } health_report { overall_health: HEALTHY } } }
I20250626 08:23:44.539973 26097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.67:52541
I20250626 08:23:44.544862 26602 heartbeater.cc:499] Master 127.22.64.126:40933 was elected leader, sending a full tablet report...
I20250626 08:23:44.547580 22785 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 08:23:44.553997 22785 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 08:23:44.557713 26240 consensus_queue.cc:237] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: NON_VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: true } }
W20250626 08:23:44.558111 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250626 08:23:44.562786 26384 raft_consensus.cc:1273] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Refusing update from remote peer fc5c7a6170284343a573042e7224d5e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250626 08:23:44.564277 26458 consensus_queue.cc:1035] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250626 08:23:44.572752 26458 raft_consensus.cc:2953] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: NON_VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: true } } }
I20250626 08:23:44.573225 26546 consensus_queue.cc:237] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } }
I20250626 08:23:44.577543 26594 raft_consensus.cc:2953] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER fc5c7a6170284343a573042e7224d5e8 (127.22.64.65) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } } }
I20250626 08:23:44.575721 26384 raft_consensus.cc:2953] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: NON_VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: true } } }
I20250626 08:23:44.586366 26084 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ae1749c72e1c46f3b26e946563289089 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 3)
W20250626 08:23:44.592931 26173 consensus_peers.cc:487] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 -> Peer a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135): Couldn't send request to peer a47f6e68e2e3439f895bc95cdec80597. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae1749c72e1c46f3b26e946563289089. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:44.592563 26098 catalog_manager.cc:5582] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 reported cstate change: config changed from index 10 to 11, NON_VOTER a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67) added. New cstate: current_term: 2 leader_uuid: "fc5c7a6170284343a573042e7224d5e8" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: NON_VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:44.596060 26084 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bb71af2b445a4783958bf42caa5b7d6b with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 08:23:44.600867 26097 catalog_manager.cc:5582] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 reported cstate change: config changed from index -1 to 8, NON_VOTER fc5c7a6170284343a573042e7224d5e8 (127.22.64.65) added. New cstate: current_term: 2 leader_uuid: "a47f6e68e2e3439f895bc95cdec80597" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 08:23:44.619807 26479 consensus_peers.cc:487] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 -> Peer fc5c7a6170284343a573042e7224d5e8 (127.22.64.65:37235): Couldn't send request to peer fc5c7a6170284343a573042e7224d5e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb71af2b445a4783958bf42caa5b7d6b. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:44.621027 26546 consensus_queue.cc:237] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: NON_VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: true } }
I20250626 08:23:44.625195 26594 raft_consensus.cc:2953] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 530bcc5617084355ac56c210c23445fc (127.22.64.66) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: NON_VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: true } } }
W20250626 08:23:44.630718 26479 consensus_peers.cc:487] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 -> Peer fc5c7a6170284343a573042e7224d5e8 (127.22.64.65:37235): Couldn't send request to peer fc5c7a6170284343a573042e7224d5e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb71af2b445a4783958bf42caa5b7d6b. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:44.636281 26084 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bb71af2b445a4783958bf42caa5b7d6b with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 08:23:44.640797 26097 catalog_manager.cc:5582] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 reported cstate change: config changed from index 8 to 9, NON_VOTER 530bcc5617084355ac56c210c23445fc (127.22.64.66) added. New cstate: current_term: 2 leader_uuid: "a47f6e68e2e3439f895bc95cdec80597" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: NON_VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 08:23:44.645097 26478 consensus_peers.cc:487] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 -> Peer 530bcc5617084355ac56c210c23445fc (127.22.64.66:43107): Couldn't send request to peer 530bcc5617084355ac56c210c23445fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb71af2b445a4783958bf42caa5b7d6b. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:45.078475 26614 ts_tablet_manager.cc:927] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: Initiating tablet copy from peer a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135)
I20250626 08:23:45.079771 26614 tablet_copy_client.cc:323] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: tablet copy: Beginning tablet copy session from remote peer at address 127.22.64.67:34135
I20250626 08:23:45.090058 26572 tablet_copy_service.cc:140] P a47f6e68e2e3439f895bc95cdec80597: Received BeginTabletCopySession request for tablet bb71af2b445a4783958bf42caa5b7d6b from peer 530bcc5617084355ac56c210c23445fc ({username='slave'} at 127.22.64.66:33813)
I20250626 08:23:45.090628 26572 tablet_copy_service.cc:161] P a47f6e68e2e3439f895bc95cdec80597: Beginning new tablet copy session on tablet bb71af2b445a4783958bf42caa5b7d6b from peer 530bcc5617084355ac56c210c23445fc at {username='slave'} at 127.22.64.66:33813: session id = 530bcc5617084355ac56c210c23445fc-bb71af2b445a4783958bf42caa5b7d6b
I20250626 08:23:45.095238 26572 tablet_copy_source_session.cc:215] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:45.098209 26614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb71af2b445a4783958bf42caa5b7d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:45.107908 26614 tablet_copy_client.cc:806] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:45.108335 26614 tablet_copy_client.cc:670] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:45.111912 26614 tablet_copy_client.cc:538] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:45.117251 26614 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: Bootstrap starting.
I20250626 08:23:45.123155 26619 ts_tablet_manager.cc:927] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: Initiating tablet copy from peer fc5c7a6170284343a573042e7224d5e8 (127.22.64.65:37235)
I20250626 08:23:45.125183 26619 tablet_copy_client.cc:323] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: tablet copy: Beginning tablet copy session from remote peer at address 127.22.64.65:37235
I20250626 08:23:45.126827 26260 tablet_copy_service.cc:140] P fc5c7a6170284343a573042e7224d5e8: Received BeginTabletCopySession request for tablet ae1749c72e1c46f3b26e946563289089 from peer a47f6e68e2e3439f895bc95cdec80597 ({username='slave'} at 127.22.64.67:32851)
I20250626 08:23:45.127367 26260 tablet_copy_service.cc:161] P fc5c7a6170284343a573042e7224d5e8: Beginning new tablet copy session on tablet ae1749c72e1c46f3b26e946563289089 from peer a47f6e68e2e3439f895bc95cdec80597 at {username='slave'} at 127.22.64.67:32851: session id = a47f6e68e2e3439f895bc95cdec80597-ae1749c72e1c46f3b26e946563289089
I20250626 08:23:45.134603 26260 tablet_copy_source_session.cc:215] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:45.137776 26619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1749c72e1c46f3b26e946563289089. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:45.151995 26619 tablet_copy_client.cc:806] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:45.152558 26619 tablet_copy_client.cc:670] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:45.156891 26619 tablet_copy_client.cc:538] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:45.162683 26619 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: Bootstrap starting.
I20250626 08:23:45.196435 26614 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:45.197074 26614 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: Bootstrap complete.
I20250626 08:23:45.197530 26614 ts_tablet_manager.cc:1397] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: Time spent bootstrapping tablet: real 0.080s user 0.067s sys 0.013s
I20250626 08:23:45.199004 26614 raft_consensus.cc:357] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: NON_VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: true } }
I20250626 08:23:45.199452 26614 raft_consensus.cc:738] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 530bcc5617084355ac56c210c23445fc, State: Initialized, Role: LEARNER
I20250626 08:23:45.199894 26614 consensus_queue.cc:260] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: NON_VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: true } }
I20250626 08:23:45.201949 26614 ts_tablet_manager.cc:1428] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: Time spent starting tablet: real 0.004s user 0.006s sys 0.001s
I20250626 08:23:45.203794 26572 tablet_copy_service.cc:342] P a47f6e68e2e3439f895bc95cdec80597: Request end of tablet copy session 530bcc5617084355ac56c210c23445fc-bb71af2b445a4783958bf42caa5b7d6b received from {username='slave'} at 127.22.64.66:33813
I20250626 08:23:45.204221 26572 tablet_copy_service.cc:434] P a47f6e68e2e3439f895bc95cdec80597: ending tablet copy session 530bcc5617084355ac56c210c23445fc-bb71af2b445a4783958bf42caa5b7d6b on tablet bb71af2b445a4783958bf42caa5b7d6b with peer 530bcc5617084355ac56c210c23445fc
I20250626 08:23:45.207913 26624 ts_tablet_manager.cc:927] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: Initiating tablet copy from peer a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135)
I20250626 08:23:45.209671 26624 tablet_copy_client.cc:323] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: tablet copy: Beginning tablet copy session from remote peer at address 127.22.64.67:34135
I20250626 08:23:45.211501 26572 tablet_copy_service.cc:140] P a47f6e68e2e3439f895bc95cdec80597: Received BeginTabletCopySession request for tablet bb71af2b445a4783958bf42caa5b7d6b from peer fc5c7a6170284343a573042e7224d5e8 ({username='slave'} at 127.22.64.65:55737)
I20250626 08:23:45.212028 26572 tablet_copy_service.cc:161] P a47f6e68e2e3439f895bc95cdec80597: Beginning new tablet copy session on tablet bb71af2b445a4783958bf42caa5b7d6b from peer fc5c7a6170284343a573042e7224d5e8 at {username='slave'} at 127.22.64.65:55737: session id = fc5c7a6170284343a573042e7224d5e8-bb71af2b445a4783958bf42caa5b7d6b
I20250626 08:23:45.219237 26572 tablet_copy_source_session.cc:215] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:45.222743 26624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb71af2b445a4783958bf42caa5b7d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:45.234416 26624 tablet_copy_client.cc:806] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:45.235072 26624 tablet_copy_client.cc:670] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:45.238718 26624 tablet_copy_client.cc:538] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:45.244349 26624 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: Bootstrap starting.
I20250626 08:23:45.257020 26619 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:45.257665 26619 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: Bootstrap complete.
I20250626 08:23:45.258111 26619 ts_tablet_manager.cc:1397] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: Time spent bootstrapping tablet: real 0.096s user 0.075s sys 0.016s
I20250626 08:23:45.259824 26619 raft_consensus.cc:357] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: NON_VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: true } }
I20250626 08:23:45.260265 26619 raft_consensus.cc:738] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a47f6e68e2e3439f895bc95cdec80597, State: Initialized, Role: LEARNER
I20250626 08:23:45.260704 26619 consensus_queue.cc:260] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: NON_VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: true } }
I20250626 08:23:45.262351 26619 ts_tablet_manager.cc:1428] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250626 08:23:45.264108 26260 tablet_copy_service.cc:342] P fc5c7a6170284343a573042e7224d5e8: Request end of tablet copy session a47f6e68e2e3439f895bc95cdec80597-ae1749c72e1c46f3b26e946563289089 received from {username='slave'} at 127.22.64.67:32851
I20250626 08:23:45.264431 26260 tablet_copy_service.cc:434] P fc5c7a6170284343a573042e7224d5e8: ending tablet copy session a47f6e68e2e3439f895bc95cdec80597-ae1749c72e1c46f3b26e946563289089 on tablet ae1749c72e1c46f3b26e946563289089 with peer a47f6e68e2e3439f895bc95cdec80597
I20250626 08:23:45.309386 26084 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ae1749c72e1c46f3b26e946563289089 with cas_config_opid_index 9: aborting the task: latest config opid_index 11; task opid_index 9
I20250626 08:23:45.321058 26624 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:45.321748 26624 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: Bootstrap complete.
I20250626 08:23:45.322204 26624 ts_tablet_manager.cc:1397] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: Time spent bootstrapping tablet: real 0.078s user 0.071s sys 0.005s
I20250626 08:23:45.324340 26624 raft_consensus.cc:357] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: NON_VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: true } }
I20250626 08:23:45.324812 26624 raft_consensus.cc:738] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: fc5c7a6170284343a573042e7224d5e8, State: Initialized, Role: LEARNER
I20250626 08:23:45.325260 26624 consensus_queue.cc:260] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: NON_VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: true } }
I20250626 08:23:45.326622 26624 ts_tablet_manager.cc:1428] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250626 08:23:45.328094 26572 tablet_copy_service.cc:342] P a47f6e68e2e3439f895bc95cdec80597: Request end of tablet copy session fc5c7a6170284343a573042e7224d5e8-bb71af2b445a4783958bf42caa5b7d6b received from {username='slave'} at 127.22.64.65:55737
I20250626 08:23:45.328501 26572 tablet_copy_service.cc:434] P a47f6e68e2e3439f895bc95cdec80597: ending tablet copy session fc5c7a6170284343a573042e7224d5e8-bb71af2b445a4783958bf42caa5b7d6b on tablet bb71af2b445a4783958bf42caa5b7d6b with peer fc5c7a6170284343a573042e7224d5e8
I20250626 08:23:45.562803 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fc5c7a6170284343a573042e7224d5e8 to finish bootstrapping
I20250626 08:23:45.576763 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 530bcc5617084355ac56c210c23445fc to finish bootstrapping
I20250626 08:23:45.579113 26546 raft_consensus.cc:1215] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250626 08:23:45.590188 26384 raft_consensus.cc:1215] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250626 08:23:45.600152 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a47f6e68e2e3439f895bc95cdec80597 to finish bootstrapping
I20250626 08:23:45.722806 26240 raft_consensus.cc:1215] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250626 08:23:45.891881 26364 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 08:23:45.893239 26526 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 08:23:45.900292 26220 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 08:23:46.037075 26630 raft_consensus.cc:1062] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: attempting to promote NON_VOTER a47f6e68e2e3439f895bc95cdec80597 to VOTER
I20250626 08:23:46.038749 26630 consensus_queue.cc:237] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } }
I20250626 08:23:46.044245 26546 raft_consensus.cc:1273] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597 [term 2 LEARNER]: Refusing update from remote peer fc5c7a6170284343a573042e7224d5e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250626 08:23:46.045846 26629 consensus_queue.cc:1035] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250626 08:23:46.046428 26384 raft_consensus.cc:1273] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Refusing update from remote peer fc5c7a6170284343a573042e7224d5e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250626 08:23:46.048117 26629 consensus_queue.cc:1035] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250626 08:23:46.056384 26629 raft_consensus.cc:2953] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } } }
I20250626 08:23:46.059243 26545 raft_consensus.cc:2953] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } } }
I20250626 08:23:46.071520 26097 catalog_manager.cc:5582] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 reported cstate change: config changed from index 11 to 12, a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "fc5c7a6170284343a573042e7224d5e8" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:46.073890 26384 raft_consensus.cc:2953] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
59fe2f40639743bdaf06dae786592327 | 127.22.64.126:40933 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.64.84:43383 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
530bcc5617084355ac56c210c23445fc | 127.22.64.66:43107 | HEALTHY | <none> | 1 | 0
a47f6e68e2e3439f895bc95cdec80597 | 127.22.64.67:34135 | HEALTHY | <none> | 1 | 0
fc5c7a6170284343a573042e7224d5e8 | 127.22.64.65:37235 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.22.64.65 | experimental | 127.22.64.65:37235
local_ip_for_outbound_sockets | 127.22.64.66 | experimental | 127.22.64.66:43107
local_ip_for_outbound_sockets | 127.22.64.67 | experimental | 127.22.64.67:34135
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.64.65:37235
server_dump_info_path | /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.64.66:43107
server_dump_info_path | /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.64.67:34135
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.64.84:43383 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250626 08:23:46.140722 22785 log_verifier.cc:126] Checking tablet 9761923646a248148bee793cbae0b5ec
I20250626 08:23:46.175323 22785 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 9761923646a248148bee793cbae0b5ec
I20250626 08:23:46.175580 22785 log_verifier.cc:126] Checking tablet ae1749c72e1c46f3b26e946563289089
I20250626 08:23:46.199152 26594 raft_consensus.cc:1062] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: attempting to promote NON_VOTER 530bcc5617084355ac56c210c23445fc to VOTER
I20250626 08:23:46.200742 26594 consensus_queue.cc:237] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:23:46.206027 26384 raft_consensus.cc:1273] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 2 LEARNER]: Refusing update from remote peer a47f6e68e2e3439f895bc95cdec80597: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250626 08:23:46.206775 26240 raft_consensus.cc:1273] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 2 LEARNER]: Refusing update from remote peer a47f6e68e2e3439f895bc95cdec80597: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250626 08:23:46.207396 26594 consensus_queue.cc:1035] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250626 08:23:46.208217 26632 consensus_queue.cc:1035] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250626 08:23:46.216470 26594 raft_consensus.cc:2953] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 530bcc5617084355ac56c210c23445fc (127.22.64.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } }
I20250626 08:23:46.218556 26384 raft_consensus.cc:2953] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 530bcc5617084355ac56c210c23445fc (127.22.64.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } }
I20250626 08:23:46.221496 26239 raft_consensus.cc:2953] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 530bcc5617084355ac56c210c23445fc (127.22.64.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } }
I20250626 08:23:46.229835 26098 catalog_manager.cc:5582] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 reported cstate change: config changed from index 9 to 10, 530bcc5617084355ac56c210c23445fc (127.22.64.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a47f6e68e2e3439f895bc95cdec80597" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:46.255200 26633 raft_consensus.cc:1062] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: attempting to promote NON_VOTER fc5c7a6170284343a573042e7224d5e8 to VOTER
I20250626 08:23:46.257213 26633 consensus_queue.cc:237] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:23:46.254719 26632 raft_consensus.cc:1062] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: attempting to promote NON_VOTER fc5c7a6170284343a573042e7224d5e8 to VOTER
I20250626 08:23:46.263855 26384 raft_consensus.cc:1273] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Refusing update from remote peer a47f6e68e2e3439f895bc95cdec80597: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
W20250626 08:23:46.264164 26632 raft_consensus.cc:1066] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Unable to promote non-voter fc5c7a6170284343a573042e7224d5e8: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: NON_VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: true } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:23:46.264334 26239 raft_consensus.cc:1273] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 2 LEARNER]: Refusing update from remote peer a47f6e68e2e3439f895bc95cdec80597: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250626 08:23:46.265355 26594 consensus_queue.cc:1035] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250626 08:23:46.266130 26670 consensus_queue.cc:1035] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250626 08:23:46.273192 26670 raft_consensus.cc:2953] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, fc5c7a6170284343a573042e7224d5e8 (127.22.64.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } }
I20250626 08:23:46.277187 26239 raft_consensus.cc:2953] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, fc5c7a6170284343a573042e7224d5e8 (127.22.64.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } }
I20250626 08:23:46.275307 26384 raft_consensus.cc:2953] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, fc5c7a6170284343a573042e7224d5e8 (127.22.64.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } }
I20250626 08:23:46.289412 26096 catalog_manager.cc:5582] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 reported cstate change: config changed from index 10 to 11, fc5c7a6170284343a573042e7224d5e8 (127.22.64.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a47f6e68e2e3439f895bc95cdec80597" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:46.296595 22785 log_verifier.cc:177] Verified matching terms for 12 ops in tablet ae1749c72e1c46f3b26e946563289089
I20250626 08:23:46.296919 22785 log_verifier.cc:126] Checking tablet bb71af2b445a4783958bf42caa5b7d6b
I20250626 08:23:46.384914 22785 log_verifier.cc:177] Verified matching terms for 11 ops in tablet bb71af2b445a4783958bf42caa5b7d6b
I20250626 08:23:46.385483 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 26064
I20250626 08:23:46.417738 22785 minidump.cc:252] Setting minidump size limit to 20M
I20250626 08:23:46.419183 22785 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:46.420641 22785 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:46.433693 26673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:46.434299 26672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:46.435250 26675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:46.537340 22785 server_base.cc:1048] running on GCE node
I20250626 08:23:46.538936 22785 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 08:23:46.539238 22785 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 08:23:46.539436 22785 hybrid_clock.cc:648] HybridClock initialized: now 1750926226539417 us; error 0 us; skew 500 ppm
I20250626 08:23:46.540297 22785 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:46.543970 22785 webserver.cc:469] Webserver started at http://0.0.0.0:33689/ using document root <none> and password file <none>
I20250626 08:23:46.545065 22785 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:46.545324 22785 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:46.551209 22785 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20250626 08:23:46.555051 26680 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:46.556066 22785 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250626 08:23:46.556378 22785 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
uuid: "59fe2f40639743bdaf06dae786592327"
format_stamp: "Formatted at 2025-06-26 08:23:24 on dist-test-slave-z1j3"
I20250626 08:23:46.558055 22785 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:46.571221 22785 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:46.572873 22785 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:46.573411 22785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:46.583003 22785 sys_catalog.cc:263] Verifying existing consensus state
W20250626 08:23:46.586689 22785 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.22.64.126:40933 exists but no master address supplied!
I20250626 08:23:46.588819 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Bootstrap starting.
I20250626 08:23:46.632213 22785 log.cc:826] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:46.699951 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:46.700767 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Bootstrap complete.
I20250626 08:23:46.714347 22785 raft_consensus.cc:357] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:46.714967 22785 raft_consensus.cc:738] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fe2f40639743bdaf06dae786592327, State: Initialized, Role: FOLLOWER
I20250626 08:23:46.715835 22785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:46.716331 22785 raft_consensus.cc:397] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:23:46.716588 22785 raft_consensus.cc:491] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:23:46.716912 22785 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:46.722759 22785 raft_consensus.cc:513] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:46.723553 22785 leader_election.cc:304] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59fe2f40639743bdaf06dae786592327; no voters:
I20250626 08:23:46.724766 22785 leader_election.cc:290] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250626 08:23:46.725080 26687 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:46.726390 26687 raft_consensus.cc:695] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 4 LEADER]: Becoming Leader. State: Replica: 59fe2f40639743bdaf06dae786592327, State: Running, Role: LEADER
I20250626 08:23:46.727180 26687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:46.734853 26688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "59fe2f40639743bdaf06dae786592327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } } }
I20250626 08:23:46.735370 26689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59fe2f40639743bdaf06dae786592327. Latest consensus state: current_term: 4 leader_uuid: "59fe2f40639743bdaf06dae786592327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } } }
I20250626 08:23:46.735641 26688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: This master's current role is: LEADER
I20250626 08:23:46.735955 26689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: This master's current role is: LEADER
I20250626 08:23:46.763247 22785 tablet_replica.cc:331] stopping tablet replica
I20250626 08:23:46.763974 22785 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 4 LEADER]: Raft consensus shutting down.
I20250626 08:23:46.764526 22785 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:46.767496 22785 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250626 08:23:46.768146 22785 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250626 08:23:46.794250 22785 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250626 08:23:47.311219 26429 heartbeater.cc:646] Failed to heartbeat to 127.22.64.126:40933 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.64.126:40933: connect: Connection refused (error 111)
W20250626 08:23:47.313752 26285 heartbeater.cc:646] Failed to heartbeat to 127.22.64.126:40933 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.64.126:40933: connect: Connection refused (error 111)
W20250626 08:23:47.321617 26602 heartbeater.cc:646] Failed to heartbeat to 127.22.64.126:40933 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.64.126:40933: connect: Connection refused (error 111)
I20250626 08:23:52.013042 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 26135
I20250626 08:23:52.040889 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 26287
I20250626 08:23:52.068482 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 26436
I20250626 08:23:52.101156 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:40933
--webserver_interface=127.22.64.126
--webserver_port=36075
--builtin_ntp_servers=127.22.64.84:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.64.126:40933 with env {}
W20250626 08:23:52.415114 26763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:52.415767 26763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:52.416254 26763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:52.447701 26763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 08:23:52.447999 26763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:52.448284 26763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 08:23:52.448563 26763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 08:23:52.484377 26763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.64.126:40933
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:40933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.64.126
--webserver_port=36075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:52.485780 26763 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:52.487598 26763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:52.504197 26770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:52.504724 26772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:52.505887 26763 server_base.cc:1048] running on GCE node
W20250626 08:23:52.504189 26769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:53.698714 26763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:53.702019 26763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:53.703523 26763 hybrid_clock.cc:648] HybridClock initialized: now 1750926233703493 us; error 54 us; skew 500 ppm
I20250626 08:23:53.704397 26763 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:53.711035 26763 webserver.cc:469] Webserver started at http://127.22.64.126:36075/ using document root <none> and password file <none>
I20250626 08:23:53.712025 26763 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:53.712234 26763 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:53.720602 26763 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20250626 08:23:53.725485 26779 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:53.726639 26763 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 08:23:53.726984 26763 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
uuid: "59fe2f40639743bdaf06dae786592327"
format_stamp: "Formatted at 2025-06-26 08:23:24 on dist-test-slave-z1j3"
I20250626 08:23:53.729032 26763 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:53.783502 26763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:53.785094 26763 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:53.785564 26763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:53.863103 26763 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.126:40933
I20250626 08:23:53.863180 26830 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.126:40933 every 8 connection(s)
I20250626 08:23:53.866089 26763 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 08:23:53.876421 26831 sys_catalog.cc:263] Verifying existing consensus state
I20250626 08:23:53.876961 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 26763
I20250626 08:23:53.879091 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.65:37235
--local_ip_for_outbound_sockets=127.22.64.65
--tserver_master_addrs=127.22.64.126:40933
--webserver_port=36379
--webserver_interface=127.22.64.65
--builtin_ntp_servers=127.22.64.84:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:23:53.884950 26831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Bootstrap starting.
I20250626 08:23:53.898615 26831 log.cc:826] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:53.977528 26831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:53.978389 26831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Bootstrap complete.
I20250626 08:23:53.997525 26831 raft_consensus.cc:357] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:53.999572 26831 raft_consensus.cc:738] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fe2f40639743bdaf06dae786592327, State: Initialized, Role: FOLLOWER
I20250626 08:23:54.000363 26831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:54.000821 26831 raft_consensus.cc:397] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:23:54.001071 26831 raft_consensus.cc:491] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:23:54.001345 26831 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:54.006618 26831 raft_consensus.cc:513] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:54.007345 26831 leader_election.cc:304] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59fe2f40639743bdaf06dae786592327; no voters:
I20250626 08:23:54.009588 26831 leader_election.cc:290] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250626 08:23:54.010128 26835 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:23:54.012813 26835 raft_consensus.cc:695] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [term 6 LEADER]: Becoming Leader. State: Replica: 59fe2f40639743bdaf06dae786592327, State: Running, Role: LEADER
I20250626 08:23:54.013836 26835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } }
I20250626 08:23:54.015338 26831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:23:54.026288 26837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "59fe2f40639743bdaf06dae786592327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } } }
I20250626 08:23:54.027364 26837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: This master's current role is: LEADER
I20250626 08:23:54.027590 26836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59fe2f40639743bdaf06dae786592327. Latest consensus state: current_term: 6 leader_uuid: "59fe2f40639743bdaf06dae786592327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fe2f40639743bdaf06dae786592327" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 40933 } } }
I20250626 08:23:54.028270 26836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327 [sys.catalog]: This master's current role is: LEADER
I20250626 08:23:54.040587 26842 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:23:54.054809 26842 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=5213ff601e694deda4b7cefeaee2ffb9]
I20250626 08:23:54.056730 26842 catalog_manager.cc:671] Loaded metadata for table TestTable [id=ca389bfe2c7842609015bece42fabab2]
I20250626 08:23:54.058633 26842 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=fd17003f89324fb3bf2e63608349bf57]
I20250626 08:23:54.068378 26842 tablet_loader.cc:96] loaded metadata for tablet 9761923646a248148bee793cbae0b5ec (table TestTable2 [id=5213ff601e694deda4b7cefeaee2ffb9])
I20250626 08:23:54.070147 26842 tablet_loader.cc:96] loaded metadata for tablet ae1749c72e1c46f3b26e946563289089 (table TestTable1 [id=fd17003f89324fb3bf2e63608349bf57])
I20250626 08:23:54.071681 26842 tablet_loader.cc:96] loaded metadata for tablet bb71af2b445a4783958bf42caa5b7d6b (table TestTable [id=ca389bfe2c7842609015bece42fabab2])
I20250626 08:23:54.073282 26842 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:23:54.079453 26842 catalog_manager.cc:1261] Loaded cluster ID: 7f9f7a6de1c14fc99e2cba320babdec2
I20250626 08:23:54.079845 26842 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:23:54.089190 26842 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:23:54.095659 26842 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 59fe2f40639743bdaf06dae786592327: Loaded TSK: 0
I20250626 08:23:54.097527 26842 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250626 08:23:54.238684 26833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:54.239203 26833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:54.239728 26833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:54.273212 26833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:54.274749 26833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.65
I20250626 08:23:54.311546 26833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.65:37235
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.64.65
--webserver_port=36379
--tserver_master_addrs=127.22.64.126:40933
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.65
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:54.312927 26833 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:54.314690 26833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:54.333003 26859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:54.336098 26833 server_base.cc:1048] running on GCE node
W20250626 08:23:54.336354 26861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:54.333967 26858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:55.526585 26833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:55.529660 26833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:55.531136 26833 hybrid_clock.cc:648] HybridClock initialized: now 1750926235531075 us; error 82 us; skew 500 ppm
I20250626 08:23:55.532013 26833 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:55.544752 26833 webserver.cc:469] Webserver started at http://127.22.64.65:36379/ using document root <none> and password file <none>
I20250626 08:23:55.545773 26833 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:55.546025 26833 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:55.554982 26833 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20250626 08:23:55.560658 26868 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:55.561900 26833 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20250626 08:23:55.562284 26833 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fc5c7a6170284343a573042e7224d5e8"
format_stamp: "Formatted at 2025-06-26 08:23:26 on dist-test-slave-z1j3"
I20250626 08:23:55.564386 26833 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:55.624414 26833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:55.625957 26833 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:55.626448 26833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:55.629145 26833 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:23:55.636042 26875 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250626 08:23:55.655153 26833 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250626 08:23:55.655465 26833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.021s user 0.002s sys 0.000s
I20250626 08:23:55.655781 26833 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250626 08:23:55.661691 26875 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Bootstrap starting.
I20250626 08:23:55.664492 26833 ts_tablet_manager.cc:610] Registered 2 tablets
I20250626 08:23:55.664780 26833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.007s sys 0.000s
I20250626 08:23:55.741084 26875 log.cc:826] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:55.896732 26875 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:55.898037 26875 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Bootstrap complete.
I20250626 08:23:55.900167 26875 ts_tablet_manager.cc:1397] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Time spent bootstrapping tablet: real 0.239s user 0.162s sys 0.063s
I20250626 08:23:55.907702 26833 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.65:37235
I20250626 08:23:55.907882 26982 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.65:37235 every 8 connection(s)
I20250626 08:23:55.910606 26833 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 08:23:55.920912 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 26833
I20250626 08:23:55.923617 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.66:43107
--local_ip_for_outbound_sockets=127.22.64.66
--tserver_master_addrs=127.22.64.126:40933
--webserver_port=46315
--webserver_interface=127.22.64.66
--builtin_ntp_servers=127.22.64.84:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:23:55.931154 26875 raft_consensus.cc:357] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } }
I20250626 08:23:55.935722 26875 raft_consensus.cc:738] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5c7a6170284343a573042e7224d5e8, State: Initialized, Role: FOLLOWER
I20250626 08:23:55.936702 26875 consensus_queue.cc:260] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } }
I20250626 08:23:55.946187 26875 ts_tablet_manager.cc:1428] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8: Time spent starting tablet: real 0.046s user 0.029s sys 0.000s
I20250626 08:23:55.947065 26875 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: Bootstrap starting.
I20250626 08:23:55.964210 26983 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40933
I20250626 08:23:55.965154 26983 heartbeater.cc:461] Registering TS with master...
I20250626 08:23:55.967597 26983 heartbeater.cc:507] Master 127.22.64.126:40933 requested a full tablet report, sending...
I20250626 08:23:55.975873 26796 ts_manager.cc:194] Registered new tserver with Master: fc5c7a6170284343a573042e7224d5e8 (127.22.64.65:37235)
I20250626 08:23:55.980579 26796 catalog_manager.cc:5582] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 reported cstate change: config changed from index -1 to 11, term changed from 0 to 2, VOTER 530bcc5617084355ac56c210c23445fc (127.22.64.66) added, VOTER a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67) added, VOTER fc5c7a6170284343a573042e7224d5e8 (127.22.64.65) added. New cstate: current_term: 2 committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } }
I20250626 08:23:56.028445 26796 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.65:45171
I20250626 08:23:56.032943 26983 heartbeater.cc:499] Master 127.22.64.126:40933 was elected leader, sending a full tablet report...
I20250626 08:23:56.049737 26875 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:56.050597 26875 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: Bootstrap complete.
I20250626 08:23:56.051831 26875 ts_tablet_manager.cc:1397] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: Time spent bootstrapping tablet: real 0.105s user 0.085s sys 0.018s
I20250626 08:23:56.053361 26875 raft_consensus.cc:357] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:23:56.053810 26875 raft_consensus.cc:738] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5c7a6170284343a573042e7224d5e8, State: Initialized, Role: FOLLOWER
I20250626 08:23:56.054330 26875 consensus_queue.cc:260] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:23:56.055855 26875 ts_tablet_manager.cc:1428] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250626 08:23:56.284340 26988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:56.284873 26988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:56.285429 26988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:56.319414 26988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:56.320308 26988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.66
I20250626 08:23:56.356401 26988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.66:43107
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.64.66
--webserver_port=46315
--tserver_master_addrs=127.22.64.126:40933
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.66
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:56.357829 26988 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:56.359606 26988 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:56.375877 26999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:57.456372 27005 raft_consensus.cc:491] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:57.456883 27005 raft_consensus.cc:513] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } }
I20250626 08:23:57.459470 27005 leader_election.cc:290] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 530bcc5617084355ac56c210c23445fc (127.22.64.66:43107), a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135)
W20250626 08:23:57.463639 26870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.64.66:43107: connect: Connection refused (error 111)
W20250626 08:23:57.467456 26871 leader_election.cc:336] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135): Network error: Client connection negotiation failed: client connection to 127.22.64.67:34135: connect: Connection refused (error 111)
W20250626 08:23:57.467943 26870 leader_election.cc:336] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 530bcc5617084355ac56c210c23445fc (127.22.64.66:43107): Network error: Client connection negotiation failed: client connection to 127.22.64.66:43107: connect: Connection refused (error 111)
I20250626 08:23:57.468380 26870 leader_election.cc:304] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc5c7a6170284343a573042e7224d5e8; no voters: 530bcc5617084355ac56c210c23445fc, a47f6e68e2e3439f895bc95cdec80597
I20250626 08:23:57.469254 27005 raft_consensus.cc:2747] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250626 08:23:56.376192 27002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:56.378463 26988 server_base.cc:1048] running on GCE node
W20250626 08:23:56.376507 27000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:57.556329 26988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:57.558694 26988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:57.560063 26988 hybrid_clock.cc:648] HybridClock initialized: now 1750926237560007 us; error 66 us; skew 500 ppm
I20250626 08:23:57.560884 26988 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:57.567438 26988 webserver.cc:469] Webserver started at http://127.22.64.66:46315/ using document root <none> and password file <none>
I20250626 08:23:57.568425 26988 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:57.568665 26988 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:57.576913 26988 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250626 08:23:57.581794 27013 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:57.583024 26988 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250626 08:23:57.583359 26988 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "530bcc5617084355ac56c210c23445fc"
format_stamp: "Formatted at 2025-06-26 08:23:28 on dist-test-slave-z1j3"
I20250626 08:23:57.585341 26988 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:57.648595 26988 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:57.650123 26988 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:57.650599 26988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:57.653194 26988 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:23:57.659137 27020 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250626 08:23:57.663584 27005 raft_consensus.cc:491] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:57.664132 27005 raft_consensus.cc:513] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:23:57.666095 27005 leader_election.cc:290] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135), 530bcc5617084355ac56c210c23445fc (127.22.64.66:43107)
W20250626 08:23:57.669595 26870 leader_election.cc:336] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 530bcc5617084355ac56c210c23445fc (127.22.64.66:43107): Network error: Client connection negotiation failed: client connection to 127.22.64.66:43107: connect: Connection refused (error 111)
W20250626 08:23:57.669979 26871 leader_election.cc:336] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135): Network error: Client connection negotiation failed: client connection to 127.22.64.67:34135: connect: Connection refused (error 111)
I20250626 08:23:57.670358 26871 leader_election.cc:304] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc5c7a6170284343a573042e7224d5e8; no voters: 530bcc5617084355ac56c210c23445fc, a47f6e68e2e3439f895bc95cdec80597
I20250626 08:23:57.670930 27005 raft_consensus.cc:2747] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:57.679466 26988 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250626 08:23:57.679735 26988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.022s user 0.002s sys 0.000s
I20250626 08:23:57.680032 26988 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250626 08:23:57.685587 27020 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: Bootstrap starting.
I20250626 08:23:57.691881 26988 ts_tablet_manager.cc:610] Registered 3 tablets
I20250626 08:23:57.692188 26988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.008s sys 0.003s
I20250626 08:23:57.755549 27020 log.cc:826] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:57.849529 27020 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:57.850785 27020 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: Bootstrap complete.
I20250626 08:23:57.852211 27020 ts_tablet_manager.cc:1397] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: Time spent bootstrapping tablet: real 0.167s user 0.134s sys 0.028s
I20250626 08:23:57.872188 27020 raft_consensus.cc:357] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } }
I20250626 08:23:57.875509 27020 raft_consensus.cc:738] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530bcc5617084355ac56c210c23445fc, State: Initialized, Role: FOLLOWER
I20250626 08:23:57.876582 27020 consensus_queue.cc:260] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } }
I20250626 08:23:57.884742 26988 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.66:43107
I20250626 08:23:57.885056 27129 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.66:43107 every 8 connection(s)
I20250626 08:23:57.886921 27020 ts_tablet_manager.cc:1428] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc: Time spent starting tablet: real 0.034s user 0.025s sys 0.009s
I20250626 08:23:57.887714 27020 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: Bootstrap starting.
I20250626 08:23:57.887739 26988 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 08:23:57.889011 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 26988
I20250626 08:23:57.891584 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.67:34135
--local_ip_for_outbound_sockets=127.22.64.67
--tserver_master_addrs=127.22.64.126:40933
--webserver_port=40387
--webserver_interface=127.22.64.67
--builtin_ntp_servers=127.22.64.84:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 08:23:57.942740 27130 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40933
I20250626 08:23:57.943231 27130 heartbeater.cc:461] Registering TS with master...
I20250626 08:23:57.944532 27130 heartbeater.cc:507] Master 127.22.64.126:40933 requested a full tablet report, sending...
I20250626 08:23:57.949112 26796 ts_manager.cc:194] Registered new tserver with Master: 530bcc5617084355ac56c210c23445fc (127.22.64.66:43107)
I20250626 08:23:57.954634 26796 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.66:35715
I20250626 08:23:58.001124 27020 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:58.001958 27020 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: Bootstrap complete.
I20250626 08:23:58.003314 27020 ts_tablet_manager.cc:1397] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: Time spent bootstrapping tablet: real 0.116s user 0.086s sys 0.024s
I20250626 08:23:58.004902 27020 raft_consensus.cc:357] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:23:58.005391 27020 raft_consensus.cc:738] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530bcc5617084355ac56c210c23445fc, State: Initialized, Role: FOLLOWER
I20250626 08:23:58.005913 27020 consensus_queue.cc:260] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:23:58.007032 27130 heartbeater.cc:499] Master 127.22.64.126:40933 was elected leader, sending a full tablet report...
I20250626 08:23:58.007313 27020 ts_tablet_manager.cc:1428] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250626 08:23:58.007905 27020 tablet_bootstrap.cc:492] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Bootstrap starting.
I20250626 08:23:58.113546 27020 tablet_bootstrap.cc:492] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:58.114323 27020 tablet_bootstrap.cc:492] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Bootstrap complete.
I20250626 08:23:58.115924 27020 ts_tablet_manager.cc:1397] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Time spent bootstrapping tablet: real 0.108s user 0.090s sys 0.016s
I20250626 08:23:58.117535 27020 raft_consensus.cc:357] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } }
I20250626 08:23:58.117908 27020 raft_consensus.cc:738] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530bcc5617084355ac56c210c23445fc, State: Initialized, Role: FOLLOWER
I20250626 08:23:58.118505 27020 consensus_queue.cc:260] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } }
I20250626 08:23:58.118911 27020 raft_consensus.cc:397] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:23:58.119163 27020 raft_consensus.cc:491] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:23:58.119448 27020 raft_consensus.cc:3058] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:58.124871 27020 raft_consensus.cc:513] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } }
I20250626 08:23:58.125566 27020 leader_election.cc:304] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 530bcc5617084355ac56c210c23445fc; no voters:
I20250626 08:23:58.126076 27020 leader_election.cc:290] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [CANDIDATE]: Term 3 election: Requested vote from peers
I20250626 08:23:58.126335 27126 raft_consensus.cc:2802] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:58.130136 27020 ts_tablet_manager.cc:1428] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20250626 08:23:58.130710 27126 raft_consensus.cc:695] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [term 3 LEADER]: Becoming Leader. State: Replica: 530bcc5617084355ac56c210c23445fc, State: Running, Role: LEADER
I20250626 08:23:58.131484 27126 consensus_queue.cc:237] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } }
I20250626 08:23:58.144932 26796 catalog_manager.cc:5582] T 9761923646a248148bee793cbae0b5ec P 530bcc5617084355ac56c210c23445fc reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "530bcc5617084355ac56c210c23445fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } health_report { overall_health: HEALTHY } } }
W20250626 08:23:58.235257 27134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:23:58.235755 27134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:23:58.236207 27134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:23:58.267689 27134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:23:58.268520 27134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.67
I20250626 08:23:58.305410 27134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.67:34135
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.64.67
--webserver_port=40387
--tserver_master_addrs=127.22.64.126:40933
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.67
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:23:58.306906 27134 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:23:58.308636 27134 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:23:58.325081 27150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:23:59.231084 27156 raft_consensus.cc:491] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:59.231810 27156 raft_consensus.cc:513] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:23:59.235527 27156 leader_election.cc:290] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135), fc5c7a6170284343a573042e7224d5e8 (127.22.64.65:37235)
W20250626 08:23:59.241925 27016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.64.67:34135: connect: Connection refused (error 111)
W20250626 08:23:59.264087 27016 leader_election.cc:336] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135): Network error: Client connection negotiation failed: client connection to 127.22.64.67:34135: connect: Connection refused (error 111)
I20250626 08:23:59.269769 26938 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" candidate_uuid: "530bcc5617084355ac56c210c23445fc" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "fc5c7a6170284343a573042e7224d5e8" is_pre_election: true
I20250626 08:23:59.270452 26938 raft_consensus.cc:2466] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530bcc5617084355ac56c210c23445fc in term 2.
I20250626 08:23:59.271798 27016 leader_election.cc:304] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530bcc5617084355ac56c210c23445fc, fc5c7a6170284343a573042e7224d5e8; no voters: a47f6e68e2e3439f895bc95cdec80597
I20250626 08:23:59.272456 27156 raft_consensus.cc:2802] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:59.272795 27156 raft_consensus.cc:491] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:59.273059 27156 raft_consensus.cc:3058] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:59.277503 27156 raft_consensus.cc:513] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:23:59.279366 27156 leader_election.cc:290] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [CANDIDATE]: Term 3 election: Requested vote from peers a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135), fc5c7a6170284343a573042e7224d5e8 (127.22.64.65:37235)
I20250626 08:23:59.281481 26938 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" candidate_uuid: "530bcc5617084355ac56c210c23445fc" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "fc5c7a6170284343a573042e7224d5e8"
I20250626 08:23:59.282112 26938 raft_consensus.cc:3058] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:59.284947 27016 leader_election.cc:336] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135): Network error: Client connection negotiation failed: client connection to 127.22.64.67:34135: connect: Connection refused (error 111)
I20250626 08:23:59.297237 26938 raft_consensus.cc:2466] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530bcc5617084355ac56c210c23445fc in term 3.
I20250626 08:23:59.299108 27016 leader_election.cc:304] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530bcc5617084355ac56c210c23445fc, fc5c7a6170284343a573042e7224d5e8; no voters: a47f6e68e2e3439f895bc95cdec80597
I20250626 08:23:59.300379 27156 raft_consensus.cc:2802] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:59.302948 27156 raft_consensus.cc:695] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 3 LEADER]: Becoming Leader. State: Replica: 530bcc5617084355ac56c210c23445fc, State: Running, Role: LEADER
I20250626 08:23:59.304241 27156 consensus_queue.cc:237] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:23:59.319099 26796 catalog_manager.cc:5582] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc reported cstate change: term changed from 2 to 3, leader changed from <none> to 530bcc5617084355ac56c210c23445fc (127.22.64.66). New cstate: current_term: 3 leader_uuid: "530bcc5617084355ac56c210c23445fc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:59.451206 27167 raft_consensus.cc:491] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:59.451776 27167 raft_consensus.cc:513] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } }
I20250626 08:23:59.454183 27167 leader_election.cc:290] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 530bcc5617084355ac56c210c23445fc (127.22.64.66:43107), a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135)
W20250626 08:23:59.484175 26871 leader_election.cc:336] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135): Network error: Client connection negotiation failed: client connection to 127.22.64.67:34135: connect: Connection refused (error 111)
I20250626 08:23:59.484875 27084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae1749c72e1c46f3b26e946563289089" candidate_uuid: "fc5c7a6170284343a573042e7224d5e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "530bcc5617084355ac56c210c23445fc" is_pre_election: true
I20250626 08:23:59.486001 27084 raft_consensus.cc:2466] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc5c7a6170284343a573042e7224d5e8 in term 2.
I20250626 08:23:59.488122 26870 leader_election.cc:304] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530bcc5617084355ac56c210c23445fc, fc5c7a6170284343a573042e7224d5e8; no voters: a47f6e68e2e3439f895bc95cdec80597
I20250626 08:23:59.489223 27167 raft_consensus.cc:2802] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:59.489580 27167 raft_consensus.cc:491] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:59.489888 27167 raft_consensus.cc:3058] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:59.496798 27167 raft_consensus.cc:513] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } }
I20250626 08:23:59.499990 27084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae1749c72e1c46f3b26e946563289089" candidate_uuid: "fc5c7a6170284343a573042e7224d5e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "530bcc5617084355ac56c210c23445fc"
I20250626 08:23:59.500866 27084 raft_consensus.cc:3058] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:59.503021 26871 leader_election.cc:336] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135): Network error: Client connection negotiation failed: client connection to 127.22.64.67:34135: connect: Connection refused (error 111)
I20250626 08:23:59.503607 27167 leader_election.cc:290] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 3 election: Requested vote from peers 530bcc5617084355ac56c210c23445fc (127.22.64.66:43107), a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135)
I20250626 08:23:59.511178 27084 raft_consensus.cc:2466] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc5c7a6170284343a573042e7224d5e8 in term 3.
I20250626 08:23:59.512714 26870 leader_election.cc:304] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530bcc5617084355ac56c210c23445fc, fc5c7a6170284343a573042e7224d5e8; no voters: a47f6e68e2e3439f895bc95cdec80597
I20250626 08:23:59.513783 27167 raft_consensus.cc:2802] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:59.515866 27167 raft_consensus.cc:695] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [term 3 LEADER]: Becoming Leader. State: Replica: fc5c7a6170284343a573042e7224d5e8, State: Running, Role: LEADER
I20250626 08:23:59.517064 27167 consensus_queue.cc:237] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } }
I20250626 08:23:59.528985 26796 catalog_manager.cc:5582] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "fc5c7a6170284343a573042e7224d5e8" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250626 08:23:58.325134 27151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:58.329277 27153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:23:59.613338 27152 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1286 milliseconds
I20250626 08:23:59.613452 27134 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:23:59.614738 27134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:23:59.617496 27134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:23:59.618940 27134 hybrid_clock.cc:648] HybridClock initialized: now 1750926239618900 us; error 58 us; skew 500 ppm
I20250626 08:23:59.619750 27134 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:23:59.626188 27134 webserver.cc:469] Webserver started at http://127.22.64.67:40387/ using document root <none> and password file <none>
I20250626 08:23:59.627199 27134 fs_manager.cc:362] Metadata directory not provided
I20250626 08:23:59.627401 27134 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:23:59.635391 27134 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250626 08:23:59.640607 27178 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:23:59.641747 27134 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250626 08:23:59.642050 27134 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a47f6e68e2e3439f895bc95cdec80597"
format_stamp: "Formatted at 2025-06-26 08:23:30 on dist-test-slave-z1j3"
I20250626 08:23:59.644196 27134 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:23:59.697278 27134 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:23:59.698837 27134 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:23:59.699262 27134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:23:59.701799 27134 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:23:59.707749 27185 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250626 08:23:59.720386 27134 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250626 08:23:59.720635 27134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20250626 08:23:59.720973 27134 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250626 08:23:59.726825 27185 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: Bootstrap starting.
I20250626 08:23:59.729841 27134 ts_tablet_manager.cc:610] Registered 2 tablets
I20250626 08:23:59.730050 27134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.003s sys 0.003s
I20250626 08:23:59.785298 27185 log.cc:826] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: Log is configured to *not* fsync() on all Append() calls
I20250626 08:23:59.934940 26938 raft_consensus.cc:1273] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 3 FOLLOWER]: Refusing update from remote peer 530bcc5617084355ac56c210c23445fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250626 08:23:59.947594 27084 raft_consensus.cc:1273] T ae1749c72e1c46f3b26e946563289089 P 530bcc5617084355ac56c210c23445fc [term 3 FOLLOWER]: Refusing update from remote peer fc5c7a6170284343a573042e7224d5e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250626 08:23:59.950111 27167 consensus_queue.cc:1035] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250626 08:23:59.954356 27185 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:59.953967 27270 consensus_queue.cc:1035] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20250626 08:23:59.955250 27016 consensus_peers.cc:487] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc -> Peer a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135): Couldn't send request to peer a47f6e68e2e3439f895bc95cdec80597. Status: Network error: Client connection negotiation failed: client connection to 127.22.64.67:34135: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:59.955730 27185 tablet_bootstrap.cc:492] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: Bootstrap complete.
I20250626 08:23:59.957675 27185 ts_tablet_manager.cc:1397] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: Time spent bootstrapping tablet: real 0.231s user 0.156s sys 0.040s
I20250626 08:23:59.983616 27185 raft_consensus.cc:357] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } }
I20250626 08:23:59.987169 27185 raft_consensus.cc:738] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47f6e68e2e3439f895bc95cdec80597, State: Initialized, Role: FOLLOWER
I20250626 08:23:59.988656 27185 consensus_queue.cc:260] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } attrs { promote: false } }
W20250626 08:24:00.074833 26871 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.64.67:34135, user_credentials={real_user=slave}} blocked reactor thread for 73211.4us
W20250626 08:24:00.079666 26871 consensus_peers.cc:487] T ae1749c72e1c46f3b26e946563289089 P fc5c7a6170284343a573042e7224d5e8 -> Peer a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135): Couldn't send request to peer a47f6e68e2e3439f895bc95cdec80597. Status: Network error: Client connection negotiation failed: client connection to 127.22.64.67:34135: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 08:24:00.113375 27185 ts_tablet_manager.cc:1428] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597: Time spent starting tablet: real 0.155s user 0.083s sys 0.058s
I20250626 08:24:00.114703 27185 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Bootstrap starting.
I20250626 08:24:00.121169 27084 consensus_queue.cc:237] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:24:00.128554 26938 raft_consensus.cc:1273] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 3 FOLLOWER]: Refusing update from remote peer 530bcc5617084355ac56c210c23445fc: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250626 08:24:00.130437 27269 consensus_queue.cc:1035] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250626 08:24:00.138747 27270 raft_consensus.cc:2953] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } }
I20250626 08:24:00.141113 26938 raft_consensus.cc:2953] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } }
I20250626 08:24:00.150478 26781 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet bb71af2b445a4783958bf42caa5b7d6b with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:24:00.156301 26796 catalog_manager.cc:5582] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc reported cstate change: config changed from index 11 to 13, VOTER a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67) evicted. New cstate: current_term: 3 leader_uuid: "530bcc5617084355ac56c210c23445fc" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250626 08:24:00.175431 26796 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet bb71af2b445a4783958bf42caa5b7d6b on TS a47f6e68e2e3439f895bc95cdec80597: Not found: failed to reset TS proxy: Could not find TS for UUID a47f6e68e2e3439f895bc95cdec80597
I20250626 08:24:00.180300 27084 consensus_queue.cc:237] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 11, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:24:00.183284 27269 raft_consensus.cc:2953] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER fc5c7a6170284343a573042e7224d5e8 (127.22.64.65) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } } }
I20250626 08:24:00.192471 26781 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet bb71af2b445a4783958bf42caa5b7d6b with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:24:00.199780 27134 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.67:34135
I20250626 08:24:00.198957 26795 catalog_manager.cc:5582] T bb71af2b445a4783958bf42caa5b7d6b P 530bcc5617084355ac56c210c23445fc reported cstate change: config changed from index 13 to 14, VOTER fc5c7a6170284343a573042e7224d5e8 (127.22.64.65) evicted. New cstate: current_term: 3 leader_uuid: "530bcc5617084355ac56c210c23445fc" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:24:00.200379 27310 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.67:34135 every 8 connection(s)
I20250626 08:24:00.203256 27134 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 08:24:00.208482 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 27134
W20250626 08:24:00.239495 26781 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet bb71af2b445a4783958bf42caa5b7d6b on TS a47f6e68e2e3439f895bc95cdec80597 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a47f6e68e2e3439f895bc95cdec80597
I20250626 08:24:00.253400 26918 tablet_service.cc:1515] Processing DeleteTablet for tablet bb71af2b445a4783958bf42caa5b7d6b with delete_type TABLET_DATA_TOMBSTONED (TS fc5c7a6170284343a573042e7224d5e8 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:52596
I20250626 08:24:00.258275 27311 heartbeater.cc:344] Connected to a master server at 127.22.64.126:40933
I20250626 08:24:00.258824 27311 heartbeater.cc:461] Registering TS with master...
I20250626 08:24:00.260129 27311 heartbeater.cc:507] Master 127.22.64.126:40933 requested a full tablet report, sending...
I20250626 08:24:00.263805 27318 tablet_replica.cc:331] stopping tablet replica
I20250626 08:24:00.264788 27318 raft_consensus.cc:2241] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:24:00.264834 26795 ts_manager.cc:194] Registered new tserver with Master: a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135)
I20250626 08:24:00.265479 27318 raft_consensus.cc:2270] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:24:00.269385 27318 ts_tablet_manager.cc:1905] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:24:00.269632 26795 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.67:52513
I20250626 08:24:00.271060 22785 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 08:24:00.275842 27185 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:24:00.276666 27185 tablet_bootstrap.cc:492] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Bootstrap complete.
I20250626 08:24:00.277114 22785 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 08:24:00.278173 27185 ts_tablet_manager.cc:1397] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Time spent bootstrapping tablet: real 0.164s user 0.115s sys 0.020s
I20250626 08:24:00.280395 27185 raft_consensus.cc:357] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:24:00.281072 27185 raft_consensus.cc:738] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47f6e68e2e3439f895bc95cdec80597, State: Initialized, Role: FOLLOWER
W20250626 08:24:00.281315 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250626 08:24:00.281708 27185 consensus_queue.cc:260] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a47f6e68e2e3439f895bc95cdec80597" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 34135 } } peers { permanent_uuid: "fc5c7a6170284343a573042e7224d5e8" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 37235 } attrs { promote: false } } peers { permanent_uuid: "530bcc5617084355ac56c210c23445fc" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 43107 } attrs { promote: false } }
I20250626 08:24:00.282934 27311 heartbeater.cc:499] Master 127.22.64.126:40933 was elected leader, sending a full tablet report...
I20250626 08:24:00.283546 27185 ts_tablet_manager.cc:1428] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250626 08:24:00.283751 27318 ts_tablet_manager.cc:1918] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20250626 08:24:00.284122 27318 log.cc:1199] T bb71af2b445a4783958bf42caa5b7d6b P fc5c7a6170284343a573042e7224d5e8: Deleting WAL directory at /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/wals/bb71af2b445a4783958bf42caa5b7d6b
I20250626 08:24:00.285799 26782 catalog_manager.cc:4928] TS fc5c7a6170284343a573042e7224d5e8 (127.22.64.65:37235): tablet bb71af2b445a4783958bf42caa5b7d6b (table TestTable [id=ca389bfe2c7842609015bece42fabab2]) successfully deleted
I20250626 08:24:00.300781 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet bb71af2b445a4783958bf42caa5b7d6b with delete_type TABLET_DATA_TOMBSTONED (TS a47f6e68e2e3439f895bc95cdec80597 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:33440
I20250626 08:24:00.305501 27320 tablet_replica.cc:331] stopping tablet replica
I20250626 08:24:00.306131 27320 raft_consensus.cc:2241] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:24:00.306581 27320 raft_consensus.cc:2270] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:24:00.309000 27320 ts_tablet_manager.cc:1905] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:24:00.320971 27320 ts_tablet_manager.cc:1918] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20250626 08:24:00.321285 27320 log.cc:1199] T bb71af2b445a4783958bf42caa5b7d6b P a47f6e68e2e3439f895bc95cdec80597: Deleting WAL directory at /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal/wals/bb71af2b445a4783958bf42caa5b7d6b
I20250626 08:24:00.322908 26782 catalog_manager.cc:4928] TS a47f6e68e2e3439f895bc95cdec80597 (127.22.64.67:34135): tablet bb71af2b445a4783958bf42caa5b7d6b (table TestTable [id=ca389bfe2c7842609015bece42fabab2]) successfully deleted
I20250626 08:24:00.557248 27248 raft_consensus.cc:3058] T ae1749c72e1c46f3b26e946563289089 P a47f6e68e2e3439f895bc95cdec80597 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:24:01.285609 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:02.289291 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:03.293114 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:04.296861 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:05.300611 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:06.304333 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:07.307997 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:08.312516 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:09.316635 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:10.320366 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:11.323923 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:12.327620 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:12.986678 26827 debug-util.cc:398] Leaking SignalData structure 0x7b080009ff00 after lost signal to thread 26764
W20250626 08:24:12.987524 26827 debug-util.cc:398] Leaking SignalData structure 0x7b080006f5e0 after lost signal to thread 26830
W20250626 08:24:13.332334 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:14.336361 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:15.341040 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:16.344801 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:16.357901 26979 debug-util.cc:398] Leaking SignalData structure 0x7b08000c4680 after lost signal to thread 26853
W20250626 08:24:16.358799 26979 debug-util.cc:398] Leaking SignalData structure 0x7b08000c49e0 after lost signal to thread 26982
W20250626 08:24:17.348488 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:18.352849 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 08:24:19.357017 22785 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bb71af2b445a4783958bf42caa5b7d6b: tablet_id: "bb71af2b445a4783958bf42caa5b7d6b" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250626 08:24:20.360334 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 26833
I20250626 08:24:20.396140 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 26988
I20250626 08:24:20.426774 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 27134
I20250626 08:24:20.454488 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 26763
2025-06-26T08:24:20Z chronyd exiting
I20250626 08:24:20.507261 22785 test_util.cc:183] -----------------------------------------------
I20250626 08:24:20.507489 22785 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750926137374372-22785-0
[ FAILED ] AdminCliTest.TestRebuildTables (57433 ms)
[----------] 5 tests from AdminCliTest (123063 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250626 08:24:20.511385 22785 test_util.cc:276] Using random seed: 1237918318
I20250626 08:24:20.515872 22785 ts_itest-base.cc:115] Starting cluster with:
I20250626 08:24:20.516042 22785 ts_itest-base.cc:116] --------------
I20250626 08:24:20.516161 22785 ts_itest-base.cc:117] 5 tablet servers
I20250626 08:24:20.516269 22785 ts_itest-base.cc:118] 3 replicas per TS
I20250626 08:24:20.516387 22785 ts_itest-base.cc:119] --------------
2025-06-26T08:24:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T08:24:20Z Disabled control of system clock
I20250626 08:24:20.564952 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:44417
--webserver_interface=127.22.64.126
--webserver_port=0
--builtin_ntp_servers=127.22.64.84:36171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.64.126:44417
--raft_prepare_replacement_before_eviction=true with env {}
W20250626 08:24:20.874285 27346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:24:20.874882 27346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:24:20.875293 27346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:24:20.907377 27346 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 08:24:20.907744 27346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 08:24:20.907955 27346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:24:20.908154 27346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 08:24:20.908342 27346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 08:24:20.945019 27346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:36171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.64.126:44417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:44417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.64.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:24:20.946424 27346 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:24:20.948123 27346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:24:20.965502 27353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:20.966454 27352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:20.967171 27346 server_base.cc:1048] running on GCE node
W20250626 08:24:20.965713 27355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:22.160290 27346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:24:22.163717 27346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:24:22.165185 27346 hybrid_clock.cc:648] HybridClock initialized: now 1750926262165149 us; error 51 us; skew 500 ppm
I20250626 08:24:22.166003 27346 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:24:22.173380 27346 webserver.cc:469] Webserver started at http://127.22.64.126:33949/ using document root <none> and password file <none>
I20250626 08:24:22.174387 27346 fs_manager.cc:362] Metadata directory not provided
I20250626 08:24:22.174592 27346 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:24:22.175019 27346 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:24:22.179991 27346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0adec3fc486f401a809489801eb37793"
format_stamp: "Formatted at 2025-06-26 08:24:22 on dist-test-slave-z1j3"
I20250626 08:24:22.181106 27346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0adec3fc486f401a809489801eb37793"
format_stamp: "Formatted at 2025-06-26 08:24:22 on dist-test-slave-z1j3"
I20250626 08:24:22.189019 27346 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250626 08:24:22.195075 27362 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:22.196223 27346 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250626 08:24:22.196607 27346 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0adec3fc486f401a809489801eb37793"
format_stamp: "Formatted at 2025-06-26 08:24:22 on dist-test-slave-z1j3"
I20250626 08:24:22.196951 27346 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:24:22.253764 27346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:24:22.255354 27346 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:24:22.255833 27346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:24:22.330314 27346 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.126:44417
I20250626 08:24:22.330401 27413 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.126:44417 every 8 connection(s)
I20250626 08:24:22.333043 27346 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 08:24:22.335527 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 27346
I20250626 08:24:22.336182 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 08:24:22.339201 27414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:24:22.366551 27414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793: Bootstrap starting.
I20250626 08:24:22.373216 27414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793: Neither blocks nor log segments found. Creating new log.
I20250626 08:24:22.375319 27414 log.cc:826] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793: Log is configured to *not* fsync() on all Append() calls
I20250626 08:24:22.380277 27414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793: No bootstrap required, opened a new log
I20250626 08:24:22.399557 27414 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0adec3fc486f401a809489801eb37793" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 44417 } }
I20250626 08:24:22.400302 27414 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:24:22.400596 27414 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0adec3fc486f401a809489801eb37793, State: Initialized, Role: FOLLOWER
I20250626 08:24:22.401422 27414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0adec3fc486f401a809489801eb37793" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 44417 } }
I20250626 08:24:22.402120 27414 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:24:22.402504 27414 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:24:22.402870 27414 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:24:22.407159 27414 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0adec3fc486f401a809489801eb37793" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 44417 } }
I20250626 08:24:22.407910 27414 leader_election.cc:304] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0adec3fc486f401a809489801eb37793; no voters:
I20250626 08:24:22.409698 27414 leader_election.cc:290] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:24:22.410461 27419 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:24:22.412808 27419 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [term 1 LEADER]: Becoming Leader. State: Replica: 0adec3fc486f401a809489801eb37793, State: Running, Role: LEADER
I20250626 08:24:22.413811 27419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0adec3fc486f401a809489801eb37793" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 44417 } }
I20250626 08:24:22.416690 27414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:24:22.424014 27420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0adec3fc486f401a809489801eb37793. Latest consensus state: current_term: 1 leader_uuid: "0adec3fc486f401a809489801eb37793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0adec3fc486f401a809489801eb37793" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 44417 } } }
I20250626 08:24:22.425153 27420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [sys.catalog]: This master's current role is: LEADER
I20250626 08:24:22.425858 27421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0adec3fc486f401a809489801eb37793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0adec3fc486f401a809489801eb37793" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 44417 } } }
I20250626 08:24:22.426774 27421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793 [sys.catalog]: This master's current role is: LEADER
I20250626 08:24:22.429774 27427 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:24:22.441308 27427 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:24:22.459254 27427 catalog_manager.cc:1349] Generated new cluster ID: efac04e6d3b54504949801204fddd7fe
I20250626 08:24:22.459640 27427 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:24:22.482868 27427 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 08:24:22.485002 27427 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:24:22.510191 27427 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0adec3fc486f401a809489801eb37793: Generated new TSK 0
I20250626 08:24:22.511226 27427 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 08:24:22.529179 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.65:0
--local_ip_for_outbound_sockets=127.22.64.65
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:44417
--builtin_ntp_servers=127.22.64.84:36171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250626 08:24:22.855463 27438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:24:22.856004 27438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:24:22.856527 27438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:24:22.889163 27438 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 08:24:22.889629 27438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:24:22.890533 27438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.65
I20250626 08:24:22.927109 27438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:36171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.65
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:24:22.928553 27438 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:24:22.930325 27438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:24:22.948730 27444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:24.346995 27443 debug-util.cc:398] Leaking SignalData structure 0x7b0800000a80 after lost signal to thread 27438
W20250626 08:24:24.408844 27438 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.464s user 0.487s sys 0.947s
W20250626 08:24:24.409274 27438 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.465s user 0.487s sys 0.947s
W20250626 08:24:22.949421 27445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:24.410396 27446 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1463 milliseconds
W20250626 08:24:24.411288 27447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:24.411248 27438 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:24:24.415264 27438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:24:24.418169 27438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:24:24.419728 27438 hybrid_clock.cc:648] HybridClock initialized: now 1750926264419682 us; error 43 us; skew 500 ppm
I20250626 08:24:24.420895 27438 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:24:24.429029 27438 webserver.cc:469] Webserver started at http://127.22.64.65:45083/ using document root <none> and password file <none>
I20250626 08:24:24.430384 27438 fs_manager.cc:362] Metadata directory not provided
I20250626 08:24:24.430688 27438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:24:24.431300 27438 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:24:24.438048 27438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1c09d8b1d2ad49b3a98d921915f00fa7"
format_stamp: "Formatted at 2025-06-26 08:24:24 on dist-test-slave-z1j3"
I20250626 08:24:24.439765 27438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1c09d8b1d2ad49b3a98d921915f00fa7"
format_stamp: "Formatted at 2025-06-26 08:24:24 on dist-test-slave-z1j3"
I20250626 08:24:24.449718 27438 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.005s
I20250626 08:24:24.457674 27454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:24.459022 27438 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 08:24:24.459437 27438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1c09d8b1d2ad49b3a98d921915f00fa7"
format_stamp: "Formatted at 2025-06-26 08:24:24 on dist-test-slave-z1j3"
I20250626 08:24:24.459908 27438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:24:24.536015 27438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:24:24.538102 27438 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:24:24.538759 27438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:24:24.542244 27438 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:24:24.548194 27438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:24:24.548527 27438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:24.548856 27438 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:24:24.549086 27438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:24.708539 27438 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.65:43063
I20250626 08:24:24.708616 27566 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.65:43063 every 8 connection(s)
I20250626 08:24:24.712340 27438 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 08:24:24.717283 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 27438
I20250626 08:24:24.717849 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 08:24:24.728540 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.66:0
--local_ip_for_outbound_sockets=127.22.64.66
--webserver_interface=127.22.64.66
--webserver_port=0
--tserver_master_addrs=127.22.64.126:44417
--builtin_ntp_servers=127.22.64.84:36171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250626 08:24:24.756013 27567 heartbeater.cc:344] Connected to a master server at 127.22.64.126:44417
I20250626 08:24:24.756608 27567 heartbeater.cc:461] Registering TS with master...
I20250626 08:24:24.758041 27567 heartbeater.cc:507] Master 127.22.64.126:44417 requested a full tablet report, sending...
I20250626 08:24:24.761714 27379 ts_manager.cc:194] Registered new tserver with Master: 1c09d8b1d2ad49b3a98d921915f00fa7 (127.22.64.65:43063)
I20250626 08:24:24.765009 27379 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.65:60379
W20250626 08:24:25.055480 27571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:24:25.055999 27571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:24:25.056525 27571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:24:25.087719 27571 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 08:24:25.088164 27571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:24:25.088963 27571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.66
I20250626 08:24:25.123986 27571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:36171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.64.66
--webserver_port=0
--tserver_master_addrs=127.22.64.126:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.66
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:24:25.125349 27571 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:24:25.127015 27571 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:24:25.144565 27578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:25.770493 27567 heartbeater.cc:499] Master 127.22.64.126:44417 was elected leader, sending a full tablet report...
W20250626 08:24:25.144989 27577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:25.145445 27580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:26.326578 27579 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:24:26.326638 27571 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:24:26.330936 27571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:24:26.333453 27571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:24:26.334865 27571 hybrid_clock.cc:648] HybridClock initialized: now 1750926266334836 us; error 47 us; skew 500 ppm
I20250626 08:24:26.335726 27571 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:24:26.342686 27571 webserver.cc:469] Webserver started at http://127.22.64.66:34585/ using document root <none> and password file <none>
I20250626 08:24:26.343689 27571 fs_manager.cc:362] Metadata directory not provided
I20250626 08:24:26.343945 27571 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:24:26.344420 27571 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:24:26.349010 27571 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "46c7d2e282944558ba415547be3666a4"
format_stamp: "Formatted at 2025-06-26 08:24:26 on dist-test-slave-z1j3"
I20250626 08:24:26.350207 27571 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "46c7d2e282944558ba415547be3666a4"
format_stamp: "Formatted at 2025-06-26 08:24:26 on dist-test-slave-z1j3"
I20250626 08:24:26.357658 27571 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250626 08:24:26.363699 27587 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:26.364809 27571 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250626 08:24:26.365140 27571 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "46c7d2e282944558ba415547be3666a4"
format_stamp: "Formatted at 2025-06-26 08:24:26 on dist-test-slave-z1j3"
I20250626 08:24:26.365512 27571 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:24:26.428146 27571 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:24:26.429661 27571 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:24:26.430097 27571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:24:26.432720 27571 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:24:26.437018 27571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:24:26.437219 27571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:26.437500 27571 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:24:26.437654 27571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:26.575322 27571 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.66:42195
I20250626 08:24:26.575418 27699 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.66:42195 every 8 connection(s)
I20250626 08:24:26.577884 27571 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 08:24:26.579697 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 27571
I20250626 08:24:26.580132 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 08:24:26.587787 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.67:0
--local_ip_for_outbound_sockets=127.22.64.67
--webserver_interface=127.22.64.67
--webserver_port=0
--tserver_master_addrs=127.22.64.126:44417
--builtin_ntp_servers=127.22.64.84:36171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250626 08:24:26.600628 27700 heartbeater.cc:344] Connected to a master server at 127.22.64.126:44417
I20250626 08:24:26.601101 27700 heartbeater.cc:461] Registering TS with master...
I20250626 08:24:26.602165 27700 heartbeater.cc:507] Master 127.22.64.126:44417 requested a full tablet report, sending...
I20250626 08:24:26.604444 27379 ts_manager.cc:194] Registered new tserver with Master: 46c7d2e282944558ba415547be3666a4 (127.22.64.66:42195)
I20250626 08:24:26.605705 27379 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.66:34215
W20250626 08:24:26.899888 27704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:24:26.900384 27704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:24:26.900972 27704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:24:26.932349 27704 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 08:24:26.932760 27704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:24:26.933553 27704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.67
I20250626 08:24:26.969164 27704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:36171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.64.67
--webserver_port=0
--tserver_master_addrs=127.22.64.126:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.67
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:24:26.970676 27704 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:24:26.972311 27704 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:24:26.987927 27711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:27.609112 27700 heartbeater.cc:499] Master 127.22.64.126:44417 was elected leader, sending a full tablet report...
W20250626 08:24:26.989030 27710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:26.989910 27704 server_base.cc:1048] running on GCE node
W20250626 08:24:26.989662 27713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:28.169514 27704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:24:28.171952 27704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:24:28.173331 27704 hybrid_clock.cc:648] HybridClock initialized: now 1750926268173285 us; error 56 us; skew 500 ppm
I20250626 08:24:28.174187 27704 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:24:28.184461 27704 webserver.cc:469] Webserver started at http://127.22.64.67:44897/ using document root <none> and password file <none>
I20250626 08:24:28.185427 27704 fs_manager.cc:362] Metadata directory not provided
I20250626 08:24:28.185645 27704 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:24:28.186084 27704 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:24:28.190697 27704 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "55060d539ced4254ae0d3e09c15de526"
format_stamp: "Formatted at 2025-06-26 08:24:28 on dist-test-slave-z1j3"
I20250626 08:24:28.191834 27704 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "55060d539ced4254ae0d3e09c15de526"
format_stamp: "Formatted at 2025-06-26 08:24:28 on dist-test-slave-z1j3"
I20250626 08:24:28.199646 27704 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250626 08:24:28.205397 27721 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:28.206681 27704 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 08:24:28.207010 27704 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "55060d539ced4254ae0d3e09c15de526"
format_stamp: "Formatted at 2025-06-26 08:24:28 on dist-test-slave-z1j3"
I20250626 08:24:28.207363 27704 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:24:28.259255 27704 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:24:28.260757 27704 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:24:28.261193 27704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:24:28.263798 27704 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:24:28.268002 27704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:24:28.268211 27704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:28.268491 27704 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:24:28.268648 27704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:28.411019 27704 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.67:39529
I20250626 08:24:28.411109 27833 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.67:39529 every 8 connection(s)
I20250626 08:24:28.413686 27704 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 08:24:28.418804 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 27704
I20250626 08:24:28.419566 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 08:24:28.426192 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.68:0
--local_ip_for_outbound_sockets=127.22.64.68
--webserver_interface=127.22.64.68
--webserver_port=0
--tserver_master_addrs=127.22.64.126:44417
--builtin_ntp_servers=127.22.64.84:36171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250626 08:24:28.437016 27834 heartbeater.cc:344] Connected to a master server at 127.22.64.126:44417
I20250626 08:24:28.437577 27834 heartbeater.cc:461] Registering TS with master...
I20250626 08:24:28.438678 27834 heartbeater.cc:507] Master 127.22.64.126:44417 requested a full tablet report, sending...
I20250626 08:24:28.441208 27379 ts_manager.cc:194] Registered new tserver with Master: 55060d539ced4254ae0d3e09c15de526 (127.22.64.67:39529)
I20250626 08:24:28.443183 27379 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.67:35513
W20250626 08:24:28.750316 27838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:24:28.750816 27838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:24:28.751266 27838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:24:28.782958 27838 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 08:24:28.783524 27838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:24:28.784423 27838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.68
I20250626 08:24:28.820340 27838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:36171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.64.68
--webserver_port=0
--tserver_master_addrs=127.22.64.126:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.68
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:24:28.821655 27838 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:24:28.823333 27838 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:24:28.840014 27845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:29.446924 27834 heartbeater.cc:499] Master 127.22.64.126:44417 was elected leader, sending a full tablet report...
W20250626 08:24:28.840165 27844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:28.842401 27838 server_base.cc:1048] running on GCE node
W20250626 08:24:28.840193 27847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:30.021826 27838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:24:30.024739 27838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:24:30.026297 27838 hybrid_clock.cc:648] HybridClock initialized: now 1750926270026238 us; error 63 us; skew 500 ppm
I20250626 08:24:30.027135 27838 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:24:30.034651 27838 webserver.cc:469] Webserver started at http://127.22.64.68:39717/ using document root <none> and password file <none>
I20250626 08:24:30.035629 27838 fs_manager.cc:362] Metadata directory not provided
I20250626 08:24:30.035846 27838 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:24:30.036334 27838 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:24:30.041076 27838 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "6029e5b925a342efa80624049d611db8"
format_stamp: "Formatted at 2025-06-26 08:24:30 on dist-test-slave-z1j3"
I20250626 08:24:30.042567 27838 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "6029e5b925a342efa80624049d611db8"
format_stamp: "Formatted at 2025-06-26 08:24:30 on dist-test-slave-z1j3"
I20250626 08:24:30.050467 27838 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250626 08:24:30.056499 27854 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:30.057785 27838 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250626 08:24:30.058168 27838 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6029e5b925a342efa80624049d611db8"
format_stamp: "Formatted at 2025-06-26 08:24:30 on dist-test-slave-z1j3"
I20250626 08:24:30.058573 27838 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:24:30.107746 27838 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:24:30.109268 27838 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:24:30.109720 27838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:24:30.112311 27838 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:24:30.116532 27838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:24:30.116748 27838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:30.117017 27838 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:24:30.117172 27838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:30.256481 27838 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.68:46167
I20250626 08:24:30.256601 27966 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.68:46167 every 8 connection(s)
I20250626 08:24:30.259095 27838 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250626 08:24:30.263599 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 27838
I20250626 08:24:30.264204 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250626 08:24:30.271657 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.69:0
--local_ip_for_outbound_sockets=127.22.64.69
--webserver_interface=127.22.64.69
--webserver_port=0
--tserver_master_addrs=127.22.64.126:44417
--builtin_ntp_servers=127.22.64.84:36171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250626 08:24:30.282744 27967 heartbeater.cc:344] Connected to a master server at 127.22.64.126:44417
I20250626 08:24:30.283176 27967 heartbeater.cc:461] Registering TS with master...
I20250626 08:24:30.284416 27967 heartbeater.cc:507] Master 127.22.64.126:44417 requested a full tablet report, sending...
I20250626 08:24:30.286775 27379 ts_manager.cc:194] Registered new tserver with Master: 6029e5b925a342efa80624049d611db8 (127.22.64.68:46167)
I20250626 08:24:30.288007 27379 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.68:57461
W20250626 08:24:30.589578 27971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:24:30.590111 27971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:24:30.590646 27971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:24:30.622282 27971 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 08:24:30.622721 27971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:24:30.623541 27971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.69
I20250626 08:24:30.659791 27971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:36171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.64.69
--webserver_port=0
--tserver_master_addrs=127.22.64.126:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.69
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:24:30.661175 27971 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:24:30.662901 27971 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:24:30.678809 27977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:31.291081 27967 heartbeater.cc:499] Master 127.22.64.126:44417 was elected leader, sending a full tablet report...
W20250626 08:24:30.679209 27978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:30.681331 27980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:31.891120 27979 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:24:31.891150 27971 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:24:31.896163 27971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:24:31.899034 27971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:24:31.900635 27971 hybrid_clock.cc:648] HybridClock initialized: now 1750926271900594 us; error 37 us; skew 500 ppm
I20250626 08:24:31.901580 27971 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:24:31.908488 27971 webserver.cc:469] Webserver started at http://127.22.64.69:45393/ using document root <none> and password file <none>
I20250626 08:24:31.909426 27971 fs_manager.cc:362] Metadata directory not provided
I20250626 08:24:31.909628 27971 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:24:31.910126 27971 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:24:31.914728 27971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "46097eab3be24c00a656cc8f03858df1"
format_stamp: "Formatted at 2025-06-26 08:24:31 on dist-test-slave-z1j3"
I20250626 08:24:31.915827 27971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "46097eab3be24c00a656cc8f03858df1"
format_stamp: "Formatted at 2025-06-26 08:24:31 on dist-test-slave-z1j3"
I20250626 08:24:31.923998 27971 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20250626 08:24:31.930058 27988 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:31.931202 27971 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 08:24:31.931558 27971 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "46097eab3be24c00a656cc8f03858df1"
format_stamp: "Formatted at 2025-06-26 08:24:31 on dist-test-slave-z1j3"
I20250626 08:24:31.931870 27971 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:24:31.987680 27971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:24:31.989156 27971 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:24:31.989641 27971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:24:31.992391 27971 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:24:31.996802 27971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:24:31.997005 27971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:31.997203 27971 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:24:31.997345 27971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:32.156497 27971 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.69:37055
I20250626 08:24:32.156647 28100 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.69:37055 every 8 connection(s)
I20250626 08:24:32.159308 27971 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250626 08:24:32.166798 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 27971
I20250626 08:24:32.167405 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250626 08:24:32.187518 28101 heartbeater.cc:344] Connected to a master server at 127.22.64.126:44417
I20250626 08:24:32.188050 28101 heartbeater.cc:461] Registering TS with master...
I20250626 08:24:32.189209 28101 heartbeater.cc:507] Master 127.22.64.126:44417 requested a full tablet report, sending...
I20250626 08:24:32.191495 27378 ts_manager.cc:194] Registered new tserver with Master: 46097eab3be24c00a656cc8f03858df1 (127.22.64.69:37055)
I20250626 08:24:32.193071 27378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.69:53523
I20250626 08:24:32.205735 22785 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250626 08:24:32.246974 27378 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50300:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250626 08:24:32.335350 27635 tablet_service.cc:1468] Processing CreateTablet for tablet 5a1990d0d3734aa3974ca2aa24380add (DEFAULT_TABLE table=TestTable [id=eeba8fcea3534deda794622013b63770]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:24:32.340858 27769 tablet_service.cc:1468] Processing CreateTablet for tablet 5a1990d0d3734aa3974ca2aa24380add (DEFAULT_TABLE table=TestTable [id=eeba8fcea3534deda794622013b63770]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:24:32.343014 27635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1990d0d3734aa3974ca2aa24380add. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:24:32.343186 27769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1990d0d3734aa3974ca2aa24380add. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:24:32.345104 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 5a1990d0d3734aa3974ca2aa24380add (DEFAULT_TABLE table=TestTable [id=eeba8fcea3534deda794622013b63770]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:24:32.347306 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1990d0d3734aa3974ca2aa24380add. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:24:32.378055 28120 tablet_bootstrap.cc:492] T 5a1990d0d3734aa3974ca2aa24380add P 55060d539ced4254ae0d3e09c15de526: Bootstrap starting.
I20250626 08:24:32.380450 28121 tablet_bootstrap.cc:492] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4: Bootstrap starting.
I20250626 08:24:32.382942 28122 tablet_bootstrap.cc:492] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7: Bootstrap starting.
I20250626 08:24:32.388695 28120 tablet_bootstrap.cc:654] T 5a1990d0d3734aa3974ca2aa24380add P 55060d539ced4254ae0d3e09c15de526: Neither blocks nor log segments found. Creating new log.
I20250626 08:24:32.388696 28121 tablet_bootstrap.cc:654] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4: Neither blocks nor log segments found. Creating new log.
I20250626 08:24:32.391651 28120 log.cc:826] T 5a1990d0d3734aa3974ca2aa24380add P 55060d539ced4254ae0d3e09c15de526: Log is configured to *not* fsync() on all Append() calls
I20250626 08:24:32.392846 28122 tablet_bootstrap.cc:654] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7: Neither blocks nor log segments found. Creating new log.
I20250626 08:24:32.394034 28121 log.cc:826] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4: Log is configured to *not* fsync() on all Append() calls
I20250626 08:24:32.395406 28122 log.cc:826] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7: Log is configured to *not* fsync() on all Append() calls
I20250626 08:24:32.404517 28121 tablet_bootstrap.cc:492] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4: No bootstrap required, opened a new log
I20250626 08:24:32.405105 28121 ts_tablet_manager.cc:1397] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.004s
I20250626 08:24:32.405128 28120 tablet_bootstrap.cc:492] T 5a1990d0d3734aa3974ca2aa24380add P 55060d539ced4254ae0d3e09c15de526: No bootstrap required, opened a new log
I20250626 08:24:32.405637 28120 ts_tablet_manager.cc:1397] T 5a1990d0d3734aa3974ca2aa24380add P 55060d539ced4254ae0d3e09c15de526: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.013s
I20250626 08:24:32.408047 28122 tablet_bootstrap.cc:492] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7: No bootstrap required, opened a new log
I20250626 08:24:32.408617 28122 ts_tablet_manager.cc:1397] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.006s
I20250626 08:24:32.436193 28122 raft_consensus.cc:357] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c7d2e282944558ba415547be3666a4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42195 } } peers { permanent_uuid: "55060d539ced4254ae0d3e09c15de526" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 39529 } } peers { permanent_uuid: "1c09d8b1d2ad49b3a98d921915f00fa7" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 43063 } }
I20250626 08:24:32.437229 28122 raft_consensus.cc:383] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:24:32.437577 28122 raft_consensus.cc:738] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c09d8b1d2ad49b3a98d921915f00fa7, State: Initialized, Role: FOLLOWER
I20250626 08:24:32.438681 28122 consensus_queue.cc:260] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c7d2e282944558ba415547be3666a4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42195 } } peers { permanent_uuid: "55060d539ced4254ae0d3e09c15de526" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 39529 } } peers { permanent_uuid: "1c09d8b1d2ad49b3a98d921915f00fa7" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 43063 } }
I20250626 08:24:32.440593 28121 raft_consensus.cc:357] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c7d2e282944558ba415547be3666a4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42195 } } peers { permanent_uuid: "55060d539ced4254ae0d3e09c15de526" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 39529 } } peers { permanent_uuid: "1c09d8b1d2ad49b3a98d921915f00fa7" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 43063 } }
I20250626 08:24:32.441694 28121 raft_consensus.cc:383] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:24:32.442032 28121 raft_consensus.cc:738] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c7d2e282944558ba415547be3666a4, State: Initialized, Role: FOLLOWER
I20250626 08:24:32.442981 28121 consensus_queue.cc:260] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c7d2e282944558ba415547be3666a4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42195 } } peers { permanent_uuid: "55060d539ced4254ae0d3e09c15de526" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 39529 } } peers { permanent_uuid: "1c09d8b1d2ad49b3a98d921915f00fa7" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 43063 } }
I20250626 08:24:32.459681 28121 ts_tablet_manager.cc:1428] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4: Time spent starting tablet: real 0.053s user 0.029s sys 0.005s
I20250626 08:24:32.461153 28122 ts_tablet_manager.cc:1428] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7: Time spent starting tablet: real 0.052s user 0.015s sys 0.031s
I20250626 08:24:32.465111 28120 raft_consensus.cc:357] T 5a1990d0d3734aa3974ca2aa24380add P 55060d539ced4254ae0d3e09c15de526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c7d2e282944558ba415547be3666a4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42195 } } peers { permanent_uuid: "55060d539ced4254ae0d3e09c15de526" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 39529 } } peers { permanent_uuid: "1c09d8b1d2ad49b3a98d921915f00fa7" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 43063 } }
I20250626 08:24:32.466992 28120 raft_consensus.cc:383] T 5a1990d0d3734aa3974ca2aa24380add P 55060d539ced4254ae0d3e09c15de526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:24:32.467305 28120 raft_consensus.cc:738] T 5a1990d0d3734aa3974ca2aa24380add P 55060d539ced4254ae0d3e09c15de526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55060d539ced4254ae0d3e09c15de526, State: Initialized, Role: FOLLOWER
I20250626 08:24:32.468309 28120 consensus_queue.cc:260] T 5a1990d0d3734aa3974ca2aa24380add P 55060d539ced4254ae0d3e09c15de526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c7d2e282944558ba415547be3666a4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42195 } } peers { permanent_uuid: "55060d539ced4254ae0d3e09c15de526" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 39529 } } peers { permanent_uuid: "1c09d8b1d2ad49b3a98d921915f00fa7" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 43063 } }
I20250626 08:24:32.472550 28120 ts_tablet_manager.cc:1428] T 5a1990d0d3734aa3974ca2aa24380add P 55060d539ced4254ae0d3e09c15de526: Time spent starting tablet: real 0.067s user 0.020s sys 0.014s
W20250626 08:24:32.498739 27568 tablet.cc:2378] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:24:32.573491 28127 raft_consensus.cc:491] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:32.574134 28127 raft_consensus.cc:513] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c7d2e282944558ba415547be3666a4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42195 } } peers { permanent_uuid: "55060d539ced4254ae0d3e09c15de526" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 39529 } } peers { permanent_uuid: "1c09d8b1d2ad49b3a98d921915f00fa7" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 43063 } }
I20250626 08:24:32.577021 28127 leader_election.cc:290] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55060d539ced4254ae0d3e09c15de526 (127.22.64.67:39529), 1c09d8b1d2ad49b3a98d921915f00fa7 (127.22.64.65:43063)
I20250626 08:24:32.591194 27789 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a1990d0d3734aa3974ca2aa24380add" candidate_uuid: "46c7d2e282944558ba415547be3666a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55060d539ced4254ae0d3e09c15de526" is_pre_election: true
I20250626 08:24:32.591612 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a1990d0d3734aa3974ca2aa24380add" candidate_uuid: "46c7d2e282944558ba415547be3666a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c09d8b1d2ad49b3a98d921915f00fa7" is_pre_election: true
I20250626 08:24:32.592212 27789 raft_consensus.cc:2466] T 5a1990d0d3734aa3974ca2aa24380add P 55060d539ced4254ae0d3e09c15de526 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c7d2e282944558ba415547be3666a4 in term 0.
W20250626 08:24:32.592368 27701 tablet.cc:2378] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:24:32.592486 27522 raft_consensus.cc:2466] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c7d2e282944558ba415547be3666a4 in term 0.
I20250626 08:24:32.593722 27588 leader_election.cc:304] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c7d2e282944558ba415547be3666a4, 55060d539ced4254ae0d3e09c15de526; no voters:
I20250626 08:24:32.594684 28127 raft_consensus.cc:2802] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:24:32.595029 28127 raft_consensus.cc:491] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:32.595330 28127 raft_consensus.cc:3058] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:24:32.603077 28127 raft_consensus.cc:513] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c7d2e282944558ba415547be3666a4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42195 } } peers { permanent_uuid: "55060d539ced4254ae0d3e09c15de526" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 39529 } } peers { permanent_uuid: "1c09d8b1d2ad49b3a98d921915f00fa7" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 43063 } }
I20250626 08:24:32.604918 28127 leader_election.cc:290] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [CANDIDATE]: Term 1 election: Requested vote from peers 55060d539ced4254ae0d3e09c15de526 (127.22.64.67:39529), 1c09d8b1d2ad49b3a98d921915f00fa7 (127.22.64.65:43063)
I20250626 08:24:32.605934 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a1990d0d3734aa3974ca2aa24380add" candidate_uuid: "46c7d2e282944558ba415547be3666a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c09d8b1d2ad49b3a98d921915f00fa7"
I20250626 08:24:32.605865 27789 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a1990d0d3734aa3974ca2aa24380add" candidate_uuid: "46c7d2e282944558ba415547be3666a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55060d539ced4254ae0d3e09c15de526"
I20250626 08:24:32.606424 27522 raft_consensus.cc:3058] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:24:32.606534 27789 raft_consensus.cc:3058] T 5a1990d0d3734aa3974ca2aa24380add P 55060d539ced4254ae0d3e09c15de526 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:24:32.611325 27522 raft_consensus.cc:2466] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c7d2e282944558ba415547be3666a4 in term 1.
I20250626 08:24:32.612381 27589 leader_election.cc:304] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c09d8b1d2ad49b3a98d921915f00fa7, 46c7d2e282944558ba415547be3666a4; no voters:
I20250626 08:24:32.613164 28127 raft_consensus.cc:2802] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:24:32.613322 27789 raft_consensus.cc:2466] T 5a1990d0d3734aa3974ca2aa24380add P 55060d539ced4254ae0d3e09c15de526 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c7d2e282944558ba415547be3666a4 in term 1.
I20250626 08:24:32.615336 28127 raft_consensus.cc:695] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [term 1 LEADER]: Becoming Leader. State: Replica: 46c7d2e282944558ba415547be3666a4, State: Running, Role: LEADER
I20250626 08:24:32.616426 28127 consensus_queue.cc:237] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c7d2e282944558ba415547be3666a4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42195 } } peers { permanent_uuid: "55060d539ced4254ae0d3e09c15de526" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 39529 } } peers { permanent_uuid: "1c09d8b1d2ad49b3a98d921915f00fa7" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 43063 } }
I20250626 08:24:32.629765 27377 catalog_manager.cc:5582] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46c7d2e282944558ba415547be3666a4 (127.22.64.66). New cstate: current_term: 1 leader_uuid: "46c7d2e282944558ba415547be3666a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c7d2e282944558ba415547be3666a4" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 42195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55060d539ced4254ae0d3e09c15de526" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c09d8b1d2ad49b3a98d921915f00fa7" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 43063 } health_report { overall_health: UNKNOWN } } }
W20250626 08:24:32.675983 27835 tablet.cc:2378] T 5a1990d0d3734aa3974ca2aa24380add P 55060d539ced4254ae0d3e09c15de526: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:24:32.714785 22785 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250626 08:24:32.718863 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1c09d8b1d2ad49b3a98d921915f00fa7 to finish bootstrapping
I20250626 08:24:32.735455 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 46c7d2e282944558ba415547be3666a4 to finish bootstrapping
I20250626 08:24:32.747992 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 55060d539ced4254ae0d3e09c15de526 to finish bootstrapping
I20250626 08:24:32.759646 22785 test_util.cc:276] Using random seed: 1250166599
I20250626 08:24:32.784055 22785 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250626 08:24:32.784948 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 27704
I20250626 08:24:32.817449 27522 raft_consensus.cc:1273] T 5a1990d0d3734aa3974ca2aa24380add P 1c09d8b1d2ad49b3a98d921915f00fa7 [term 1 FOLLOWER]: Refusing update from remote peer 46c7d2e282944558ba415547be3666a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 08:24:32.817936 27588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.64.67:39529: connect: Connection refused (error 111)
I20250626 08:24:32.819973 28131 consensus_queue.cc:1035] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c09d8b1d2ad49b3a98d921915f00fa7" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 43063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250626 08:24:32.829394 27588 consensus_peers.cc:487] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 -> Peer 55060d539ced4254ae0d3e09c15de526 (127.22.64.67:39529): Couldn't send request to peer 55060d539ced4254ae0d3e09c15de526. Status: Network error: Client connection negotiation failed: client connection to 127.22.64.67:39529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 08:24:33.196465 28101 heartbeater.cc:499] Master 127.22.64.126:44417 was elected leader, sending a full tablet report...
W20250626 08:24:35.080900 27588 consensus_peers.cc:487] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 -> Peer 55060d539ced4254ae0d3e09c15de526 (127.22.64.67:39529): Couldn't send request to peer 55060d539ced4254ae0d3e09c15de526. Status: Network error: Client connection negotiation failed: client connection to 127.22.64.67:39529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250626 08:24:35.800446 27502 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 08:24:35.840977 28036 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 08:24:35.846632 27902 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 08:24:35.864313 27635 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
W20250626 08:24:37.379480 27588 consensus_peers.cc:487] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 -> Peer 55060d539ced4254ae0d3e09c15de526 (127.22.64.67:39529): Couldn't send request to peer 55060d539ced4254ae0d3e09c15de526. Status: Network error: Client connection negotiation failed: client connection to 127.22.64.67:39529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250626 08:24:37.644512 27635 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 08:24:37.652113 27502 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 08:24:37.677796 27902 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 08:24:37.682538 28036 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250626 08:24:37.967115 27588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.64.67:39529: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20250626 08:24:39.813810 27588 consensus_peers.cc:487] T 5a1990d0d3734aa3974ca2aa24380add P 46c7d2e282944558ba415547be3666a4 -> Peer 55060d539ced4254ae0d3e09c15de526 (127.22.64.67:39529): Couldn't send request to peer 55060d539ced4254ae0d3e09c15de526. Status: Network error: Client connection negotiation failed: client connection to 127.22.64.67:39529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250626 08:24:40.179575 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 27438
I20250626 08:24:40.215876 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 27571
I20250626 08:24:40.259683 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 27838
I20250626 08:24:40.281494 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 27971
I20250626 08:24:40.310534 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 27346
2025-06-26T08:24:40Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (19863 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (19863 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250626 08:24:40.375207 22785 test_util.cc:276] Using random seed: 1257782152
I20250626 08:24:40.379609 22785 ts_itest-base.cc:115] Starting cluster with:
I20250626 08:24:40.379777 22785 ts_itest-base.cc:116] --------------
I20250626 08:24:40.379894 22785 ts_itest-base.cc:117] 1 tablet servers
I20250626 08:24:40.379999 22785 ts_itest-base.cc:118] 1 replicas per TS
I20250626 08:24:40.380112 22785 ts_itest-base.cc:119] --------------
2025-06-26T08:24:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T08:24:40Z Disabled control of system clock
I20250626 08:24:40.428619 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:35483
--webserver_interface=127.22.64.126
--webserver_port=0
--builtin_ntp_servers=127.22.64.84:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.64.126:35483 with env {}
W20250626 08:24:40.742091 28244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:24:40.742770 28244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:24:40.743249 28244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:24:40.774333 28244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 08:24:40.774682 28244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:24:40.774933 28244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 08:24:40.775166 28244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 08:24:40.811023 28244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.64.126:35483
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:35483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.64.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:24:40.812451 28244 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:24:40.814126 28244 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:24:40.828503 28250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:40.829818 28251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:40.830405 28244 server_base.cc:1048] running on GCE node
W20250626 08:24:40.831570 28253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:42.030927 28244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:24:42.033593 28244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:24:42.035012 28244 hybrid_clock.cc:648] HybridClock initialized: now 1750926282034963 us; error 48 us; skew 500 ppm
I20250626 08:24:42.035831 28244 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:24:42.042485 28244 webserver.cc:469] Webserver started at http://127.22.64.126:46643/ using document root <none> and password file <none>
I20250626 08:24:42.043498 28244 fs_manager.cc:362] Metadata directory not provided
I20250626 08:24:42.043741 28244 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:24:42.044265 28244 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:24:42.048879 28244 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f344b3fd65694d9aab3a2eb39a61f936"
format_stamp: "Formatted at 2025-06-26 08:24:42 on dist-test-slave-z1j3"
I20250626 08:24:42.050010 28244 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f344b3fd65694d9aab3a2eb39a61f936"
format_stamp: "Formatted at 2025-06-26 08:24:42 on dist-test-slave-z1j3"
I20250626 08:24:42.057900 28244 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250626 08:24:42.063987 28260 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:42.065188 28244 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250626 08:24:42.065539 28244 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f344b3fd65694d9aab3a2eb39a61f936"
format_stamp: "Formatted at 2025-06-26 08:24:42 on dist-test-slave-z1j3"
I20250626 08:24:42.065915 28244 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:24:42.117925 28244 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:24:42.119534 28244 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:24:42.120003 28244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:24:42.192533 28244 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.126:35483
I20250626 08:24:42.192631 28311 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.126:35483 every 8 connection(s)
I20250626 08:24:42.195361 28244 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 08:24:42.198575 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 28244
I20250626 08:24:42.199083 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 08:24:42.201081 28312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:24:42.229532 28312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936: Bootstrap starting.
I20250626 08:24:42.235507 28312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936: Neither blocks nor log segments found. Creating new log.
I20250626 08:24:42.237418 28312 log.cc:826] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936: Log is configured to *not* fsync() on all Append() calls
I20250626 08:24:42.242480 28312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936: No bootstrap required, opened a new log
I20250626 08:24:42.261401 28312 raft_consensus.cc:357] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f344b3fd65694d9aab3a2eb39a61f936" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 35483 } }
I20250626 08:24:42.262189 28312 raft_consensus.cc:383] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:24:42.262545 28312 raft_consensus.cc:738] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f344b3fd65694d9aab3a2eb39a61f936, State: Initialized, Role: FOLLOWER
I20250626 08:24:42.263409 28312 consensus_queue.cc:260] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f344b3fd65694d9aab3a2eb39a61f936" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 35483 } }
I20250626 08:24:42.264128 28312 raft_consensus.cc:397] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:24:42.264463 28312 raft_consensus.cc:491] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:24:42.264736 28312 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:24:42.268935 28312 raft_consensus.cc:513] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f344b3fd65694d9aab3a2eb39a61f936" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 35483 } }
I20250626 08:24:42.269676 28312 leader_election.cc:304] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f344b3fd65694d9aab3a2eb39a61f936; no voters:
I20250626 08:24:42.271833 28312 leader_election.cc:290] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:24:42.272662 28317 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:24:42.275182 28317 raft_consensus.cc:695] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [term 1 LEADER]: Becoming Leader. State: Replica: f344b3fd65694d9aab3a2eb39a61f936, State: Running, Role: LEADER
I20250626 08:24:42.276103 28312 sys_catalog.cc:564] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:24:42.275977 28317 consensus_queue.cc:237] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f344b3fd65694d9aab3a2eb39a61f936" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 35483 } }
I20250626 08:24:42.288616 28318 sys_catalog.cc:455] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f344b3fd65694d9aab3a2eb39a61f936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f344b3fd65694d9aab3a2eb39a61f936" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 35483 } } }
I20250626 08:24:42.289585 28318 sys_catalog.cc:458] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [sys.catalog]: This master's current role is: LEADER
I20250626 08:24:42.292241 28319 sys_catalog.cc:455] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f344b3fd65694d9aab3a2eb39a61f936. Latest consensus state: current_term: 1 leader_uuid: "f344b3fd65694d9aab3a2eb39a61f936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f344b3fd65694d9aab3a2eb39a61f936" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 35483 } } }
I20250626 08:24:42.293047 28319 sys_catalog.cc:458] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936 [sys.catalog]: This master's current role is: LEADER
I20250626 08:24:42.294486 28326 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:24:42.305660 28326 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:24:42.324283 28326 catalog_manager.cc:1349] Generated new cluster ID: 952b3c5ac11e4b639a0fedeec86b5c9c
I20250626 08:24:42.324648 28326 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:24:42.337505 28326 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 08:24:42.339686 28326 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:24:42.358814 28326 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f344b3fd65694d9aab3a2eb39a61f936: Generated new TSK 0
I20250626 08:24:42.360001 28326 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 08:24:42.382863 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.65:0
--local_ip_for_outbound_sockets=127.22.64.65
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:35483
--builtin_ntp_servers=127.22.64.84:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 08:24:42.701164 28337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:24:42.701730 28337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:24:42.702322 28337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:24:42.735183 28337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:24:42.736054 28337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.65
I20250626 08:24:42.771865 28337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:35483
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.65
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:24:42.773252 28337 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:24:42.774971 28337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:24:42.793687 28346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:42.794632 28344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:42.793802 28343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:44.011933 28345 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1216 milliseconds
I20250626 08:24:44.012042 28337 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:24:44.013307 28337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:24:44.015821 28337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:24:44.017200 28337 hybrid_clock.cc:648] HybridClock initialized: now 1750926284017171 us; error 45 us; skew 500 ppm
I20250626 08:24:44.018026 28337 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:24:44.025568 28337 webserver.cc:469] Webserver started at http://127.22.64.65:39069/ using document root <none> and password file <none>
I20250626 08:24:44.026611 28337 fs_manager.cc:362] Metadata directory not provided
I20250626 08:24:44.026849 28337 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:24:44.027329 28337 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:24:44.033838 28337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0f36173836d240b699135cb37f4f315f"
format_stamp: "Formatted at 2025-06-26 08:24:44 on dist-test-slave-z1j3"
I20250626 08:24:44.035108 28337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0f36173836d240b699135cb37f4f315f"
format_stamp: "Formatted at 2025-06-26 08:24:44 on dist-test-slave-z1j3"
I20250626 08:24:44.043711 28337 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20250626 08:24:44.050146 28353 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:44.051384 28337 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 08:24:44.051712 28337 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0f36173836d240b699135cb37f4f315f"
format_stamp: "Formatted at 2025-06-26 08:24:44 on dist-test-slave-z1j3"
I20250626 08:24:44.052050 28337 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:24:44.104022 28337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:24:44.105588 28337 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:24:44.106038 28337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:24:44.109186 28337 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:24:44.113502 28337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:24:44.113708 28337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:44.113988 28337 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:24:44.114149 28337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:44.279726 28337 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.65:45101
I20250626 08:24:44.279807 28465 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.65:45101 every 8 connection(s)
I20250626 08:24:44.283454 28337 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 08:24:44.291811 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 28337
I20250626 08:24:44.292343 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750926137374372-22785-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 08:24:44.307327 28466 heartbeater.cc:344] Connected to a master server at 127.22.64.126:35483
I20250626 08:24:44.307879 28466 heartbeater.cc:461] Registering TS with master...
I20250626 08:24:44.309216 28466 heartbeater.cc:507] Master 127.22.64.126:35483 requested a full tablet report, sending...
I20250626 08:24:44.312264 28277 ts_manager.cc:194] Registered new tserver with Master: 0f36173836d240b699135cb37f4f315f (127.22.64.65:45101)
I20250626 08:24:44.315047 28277 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.65:39957
I20250626 08:24:44.326973 22785 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250626 08:24:44.361380 28277 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43590:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250626 08:24:44.429553 28401 tablet_service.cc:1468] Processing CreateTablet for tablet dcef3871f0ed49a484e0aa78ede59f1b (DEFAULT_TABLE table=TestTable [id=78aad2d5d6014201ba029df71234dd80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:24:44.431414 28401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcef3871f0ed49a484e0aa78ede59f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:24:44.452149 28481 tablet_bootstrap.cc:492] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f: Bootstrap starting.
I20250626 08:24:44.458250 28481 tablet_bootstrap.cc:654] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f: Neither blocks nor log segments found. Creating new log.
I20250626 08:24:44.460043 28481 log.cc:826] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f: Log is configured to *not* fsync() on all Append() calls
I20250626 08:24:44.464666 28481 tablet_bootstrap.cc:492] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f: No bootstrap required, opened a new log
I20250626 08:24:44.465073 28481 ts_tablet_manager.cc:1397] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20250626 08:24:44.484843 28481 raft_consensus.cc:357] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f36173836d240b699135cb37f4f315f" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 45101 } }
I20250626 08:24:44.485491 28481 raft_consensus.cc:383] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:24:44.485757 28481 raft_consensus.cc:738] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f36173836d240b699135cb37f4f315f, State: Initialized, Role: FOLLOWER
I20250626 08:24:44.486464 28481 consensus_queue.cc:260] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f36173836d240b699135cb37f4f315f" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 45101 } }
I20250626 08:24:44.486965 28481 raft_consensus.cc:397] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:24:44.487244 28481 raft_consensus.cc:491] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:24:44.487533 28481 raft_consensus.cc:3058] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:24:44.491835 28481 raft_consensus.cc:513] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f36173836d240b699135cb37f4f315f" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 45101 } }
I20250626 08:24:44.492513 28481 leader_election.cc:304] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f36173836d240b699135cb37f4f315f; no voters:
I20250626 08:24:44.494397 28481 leader_election.cc:290] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:24:44.495290 28483 raft_consensus.cc:2802] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:24:44.497736 28483 raft_consensus.cc:695] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f [term 1 LEADER]: Becoming Leader. State: Replica: 0f36173836d240b699135cb37f4f315f, State: Running, Role: LEADER
I20250626 08:24:44.498421 28466 heartbeater.cc:499] Master 127.22.64.126:35483 was elected leader, sending a full tablet report...
I20250626 08:24:44.498922 28483 consensus_queue.cc:237] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f36173836d240b699135cb37f4f315f" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 45101 } }
I20250626 08:24:44.502182 28481 ts_tablet_manager.cc:1428] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f: Time spent starting tablet: real 0.037s user 0.033s sys 0.004s
I20250626 08:24:44.515205 28277 catalog_manager.cc:5582] T dcef3871f0ed49a484e0aa78ede59f1b P 0f36173836d240b699135cb37f4f315f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f36173836d240b699135cb37f4f315f (127.22.64.65). New cstate: current_term: 1 leader_uuid: "0f36173836d240b699135cb37f4f315f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f36173836d240b699135cb37f4f315f" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 45101 } health_report { overall_health: HEALTHY } } }
I20250626 08:24:44.559135 22785 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250626 08:24:44.562871 22785 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0f36173836d240b699135cb37f4f315f to finish bootstrapping
I20250626 08:24:47.305486 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 28337
I20250626 08:24:47.331614 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 28244
2025-06-26T08:24:47Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7014 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7014 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250626 08:24:47.389779 22785 test_util.cc:276] Using random seed: 1264796725
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (14 ms)
[----------] 1 test from ListTableCliParamTest (14 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-26T08:24:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T08:24:47Z Disabled control of system clock
I20250626 08:24:47.449389 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:36125
--webserver_interface=127.22.64.126
--webserver_port=0
--builtin_ntp_servers=127.22.64.84:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.64.126:36125 with env {}
W20250626 08:24:47.765893 28509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:24:47.766551 28509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:24:47.767045 28509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:24:47.801653 28509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 08:24:47.801995 28509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:24:47.802301 28509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 08:24:47.802575 28509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 08:24:47.838088 28509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.64.126:36125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:36125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.64.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:24:47.839541 28509 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:24:47.841491 28509 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:24:47.856384 28517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:47.856424 28519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:47.858512 28509 server_base.cc:1048] running on GCE node
W20250626 08:24:47.857635 28516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:49.050061 28509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:24:49.052772 28509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:24:49.054143 28509 hybrid_clock.cc:648] HybridClock initialized: now 1750926289054103 us; error 55 us; skew 500 ppm
I20250626 08:24:49.055022 28509 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:24:49.066710 28509 webserver.cc:469] Webserver started at http://127.22.64.126:37167/ using document root <none> and password file <none>
I20250626 08:24:49.067729 28509 fs_manager.cc:362] Metadata directory not provided
I20250626 08:24:49.067970 28509 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:24:49.068454 28509 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:24:49.074692 28509 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data/instance:
uuid: "7bc020c9aca84a86a6b5ee49e2e73648"
format_stamp: "Formatted at 2025-06-26 08:24:49 on dist-test-slave-z1j3"
I20250626 08:24:49.075852 28509 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal/instance:
uuid: "7bc020c9aca84a86a6b5ee49e2e73648"
format_stamp: "Formatted at 2025-06-26 08:24:49 on dist-test-slave-z1j3"
I20250626 08:24:49.083245 28509 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250626 08:24:49.089095 28526 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:49.090204 28509 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250626 08:24:49.090622 28509 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal
uuid: "7bc020c9aca84a86a6b5ee49e2e73648"
format_stamp: "Formatted at 2025-06-26 08:24:49 on dist-test-slave-z1j3"
I20250626 08:24:49.090962 28509 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:24:49.142915 28509 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:24:49.144434 28509 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:24:49.144884 28509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:24:49.216176 28509 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.126:36125
I20250626 08:24:49.216250 28577 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.126:36125 every 8 connection(s)
I20250626 08:24:49.218956 28509 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data/info.pb
I20250626 08:24:49.222455 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 28509
I20250626 08:24:49.223217 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal/instance
I20250626 08:24:49.225198 28578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:24:49.252135 28578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648: Bootstrap starting.
I20250626 08:24:49.257722 28578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648: Neither blocks nor log segments found. Creating new log.
I20250626 08:24:49.259464 28578 log.cc:826] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648: Log is configured to *not* fsync() on all Append() calls
I20250626 08:24:49.264029 28578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648: No bootstrap required, opened a new log
I20250626 08:24:49.281726 28578 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc020c9aca84a86a6b5ee49e2e73648" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 36125 } }
I20250626 08:24:49.282461 28578 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:24:49.282703 28578 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc020c9aca84a86a6b5ee49e2e73648, State: Initialized, Role: FOLLOWER
I20250626 08:24:49.283375 28578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc020c9aca84a86a6b5ee49e2e73648" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 36125 } }
I20250626 08:24:49.283882 28578 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:24:49.284144 28578 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:24:49.284442 28578 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:24:49.288975 28578 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc020c9aca84a86a6b5ee49e2e73648" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 36125 } }
I20250626 08:24:49.289705 28578 leader_election.cc:304] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bc020c9aca84a86a6b5ee49e2e73648; no voters:
I20250626 08:24:49.291505 28578 leader_election.cc:290] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:24:49.292215 28583 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:24:49.294718 28583 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [term 1 LEADER]: Becoming Leader. State: Replica: 7bc020c9aca84a86a6b5ee49e2e73648, State: Running, Role: LEADER
I20250626 08:24:49.295548 28578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:24:49.295534 28583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc020c9aca84a86a6b5ee49e2e73648" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 36125 } }
I20250626 08:24:49.307026 28584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bc020c9aca84a86a6b5ee49e2e73648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc020c9aca84a86a6b5ee49e2e73648" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 36125 } } }
I20250626 08:24:49.307307 28585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bc020c9aca84a86a6b5ee49e2e73648. Latest consensus state: current_term: 1 leader_uuid: "7bc020c9aca84a86a6b5ee49e2e73648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc020c9aca84a86a6b5ee49e2e73648" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 36125 } } }
I20250626 08:24:49.307761 28584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [sys.catalog]: This master's current role is: LEADER
I20250626 08:24:49.307835 28585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648 [sys.catalog]: This master's current role is: LEADER
I20250626 08:24:49.317581 28592 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:24:49.329186 28592 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:24:49.345129 28592 catalog_manager.cc:1349] Generated new cluster ID: eca55854cb8246bebc0bda54729738e7
I20250626 08:24:49.345443 28592 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:24:49.376237 28592 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 08:24:49.377758 28592 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:24:49.392889 28592 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7bc020c9aca84a86a6b5ee49e2e73648: Generated new TSK 0
I20250626 08:24:49.393846 28592 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 08:24:49.403164 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.65:0
--local_ip_for_outbound_sockets=127.22.64.65
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:36125
--builtin_ntp_servers=127.22.64.84:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250626 08:24:49.711402 28602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:24:49.711933 28602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:24:49.712466 28602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:24:49.743479 28602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:24:49.744350 28602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.65
I20250626 08:24:49.779655 28602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.64.65
--webserver_port=0
--tserver_master_addrs=127.22.64.126:36125
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.65
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:24:49.781013 28602 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:24:49.782780 28602 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:24:49.802330 28608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:51.200703 28607 debug-util.cc:398] Leaking SignalData structure 0x7b0800000a80 after lost signal to thread 28602
W20250626 08:24:51.269390 28602 thread.cc:640] GCE (cloud detector) Time spent creating pthread: real 1.474s user 0.576s sys 0.895s
W20250626 08:24:49.802389 28609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:51.269768 28602 thread.cc:606] GCE (cloud detector) Time spent starting thread: real 1.475s user 0.577s sys 0.895s
W20250626 08:24:51.279042 28613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:51.279467 28602 server_base.cc:1048] running on GCE node
I20250626 08:24:51.280582 28602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:24:51.282702 28602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:24:51.284070 28602 hybrid_clock.cc:648] HybridClock initialized: now 1750926291284038 us; error 33 us; skew 500 ppm
I20250626 08:24:51.284953 28602 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:24:51.291810 28602 webserver.cc:469] Webserver started at http://127.22.64.65:39355/ using document root <none> and password file <none>
I20250626 08:24:51.292757 28602 fs_manager.cc:362] Metadata directory not provided
I20250626 08:24:51.292982 28602 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:24:51.293432 28602 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:24:51.297999 28602 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data/instance:
uuid: "11d564fdd08441bb9df6532ecb8cd609"
format_stamp: "Formatted at 2025-06-26 08:24:51 on dist-test-slave-z1j3"
I20250626 08:24:51.299250 28602 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/wal/instance:
uuid: "11d564fdd08441bb9df6532ecb8cd609"
format_stamp: "Formatted at 2025-06-26 08:24:51 on dist-test-slave-z1j3"
I20250626 08:24:51.306957 28602 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250626 08:24:51.312835 28618 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:51.313966 28602 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250626 08:24:51.314322 28602 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/wal
uuid: "11d564fdd08441bb9df6532ecb8cd609"
format_stamp: "Formatted at 2025-06-26 08:24:51 on dist-test-slave-z1j3"
I20250626 08:24:51.314687 28602 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:24:51.392913 28602 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:24:51.394502 28602 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:24:51.394953 28602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:24:51.398077 28602 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:24:51.402501 28602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:24:51.402704 28602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:51.402972 28602 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:24:51.403126 28602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:51.579850 28602 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.65:44937
I20250626 08:24:51.579960 28730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.65:44937 every 8 connection(s)
I20250626 08:24:51.582633 28602 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data/info.pb
I20250626 08:24:51.584928 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 28602
I20250626 08:24:51.585479 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/wal/instance
I20250626 08:24:51.593333 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.66:0
--local_ip_for_outbound_sockets=127.22.64.66
--webserver_interface=127.22.64.66
--webserver_port=0
--tserver_master_addrs=127.22.64.126:36125
--builtin_ntp_servers=127.22.64.84:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 08:24:51.626087 28731 heartbeater.cc:344] Connected to a master server at 127.22.64.126:36125
I20250626 08:24:51.626664 28731 heartbeater.cc:461] Registering TS with master...
I20250626 08:24:51.627997 28731 heartbeater.cc:507] Master 127.22.64.126:36125 requested a full tablet report, sending...
I20250626 08:24:51.631342 28543 ts_manager.cc:194] Registered new tserver with Master: 11d564fdd08441bb9df6532ecb8cd609 (127.22.64.65:44937)
I20250626 08:24:51.634263 28543 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.65:48193
W20250626 08:24:51.943189 28735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:24:51.943701 28735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:24:51.944159 28735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:24:51.975970 28735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:24:51.976791 28735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.66
I20250626 08:24:52.012248 28735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.64.66
--webserver_port=0
--tserver_master_addrs=127.22.64.126:36125
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.66
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:24:52.013608 28735 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:24:52.015266 28735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:24:52.032117 28744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:52.638751 28731 heartbeater.cc:499] Master 127.22.64.126:36125 was elected leader, sending a full tablet report...
W20250626 08:24:52.032107 28742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:52.032451 28741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:52.034660 28735 server_base.cc:1048] running on GCE node
I20250626 08:24:53.207420 28735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:24:53.209748 28735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:24:53.211130 28735 hybrid_clock.cc:648] HybridClock initialized: now 1750926293211094 us; error 40 us; skew 500 ppm
I20250626 08:24:53.212076 28735 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:24:53.219820 28735 webserver.cc:469] Webserver started at http://127.22.64.66:44189/ using document root <none> and password file <none>
I20250626 08:24:53.221088 28735 fs_manager.cc:362] Metadata directory not provided
I20250626 08:24:53.221365 28735 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:24:53.221958 28735 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:24:53.228668 28735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data/instance:
uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c"
format_stamp: "Formatted at 2025-06-26 08:24:53 on dist-test-slave-z1j3"
I20250626 08:24:53.230314 28735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/wal/instance:
uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c"
format_stamp: "Formatted at 2025-06-26 08:24:53 on dist-test-slave-z1j3"
I20250626 08:24:53.239666 28735 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20250626 08:24:53.246610 28751 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:53.247675 28735 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250626 08:24:53.247995 28735 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/wal
uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c"
format_stamp: "Formatted at 2025-06-26 08:24:53 on dist-test-slave-z1j3"
I20250626 08:24:53.248334 28735 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:24:53.295687 28735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:24:53.297137 28735 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:24:53.297627 28735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:24:53.300204 28735 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:24:53.304906 28735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:24:53.305121 28735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:53.305321 28735 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:24:53.305473 28735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:53.443653 28735 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.66:36331
I20250626 08:24:53.443773 28863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.66:36331 every 8 connection(s)
I20250626 08:24:53.446184 28735 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data/info.pb
I20250626 08:24:53.447185 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 28735
I20250626 08:24:53.447654 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/wal/instance
I20250626 08:24:53.456103 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.67:0
--local_ip_for_outbound_sockets=127.22.64.67
--webserver_interface=127.22.64.67
--webserver_port=0
--tserver_master_addrs=127.22.64.126:36125
--builtin_ntp_servers=127.22.64.84:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 08:24:53.471822 28864 heartbeater.cc:344] Connected to a master server at 127.22.64.126:36125
I20250626 08:24:53.472262 28864 heartbeater.cc:461] Registering TS with master...
I20250626 08:24:53.473325 28864 heartbeater.cc:507] Master 127.22.64.126:36125 requested a full tablet report, sending...
I20250626 08:24:53.475669 28543 ts_manager.cc:194] Registered new tserver with Master: 55d56dd9e3c54186a39d8a9a59f8cc9c (127.22.64.66:36331)
I20250626 08:24:53.477533 28543 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.66:55811
W20250626 08:24:53.774406 28868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:24:53.774938 28868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:24:53.775468 28868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:24:53.811204 28868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:24:53.812043 28868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.67
I20250626 08:24:53.847170 28868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.64.67
--webserver_port=0
--tserver_master_addrs=127.22.64.126:36125
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.67
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:24:53.848490 28868 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:24:53.850260 28868 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:24:53.868160 28877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:24:54.481950 28864 heartbeater.cc:499] Master 127.22.64.126:36125 was elected leader, sending a full tablet report...
W20250626 08:24:53.868216 28875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:53.871008 28874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:24:55.034144 28876 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:24:55.034170 28868 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:24:55.038885 28868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:24:55.041136 28868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:24:55.042474 28868 hybrid_clock.cc:648] HybridClock initialized: now 1750926295042432 us; error 63 us; skew 500 ppm
I20250626 08:24:55.043269 28868 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:24:55.055982 28868 webserver.cc:469] Webserver started at http://127.22.64.67:44025/ using document root <none> and password file <none>
I20250626 08:24:55.056954 28868 fs_manager.cc:362] Metadata directory not provided
I20250626 08:24:55.057183 28868 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:24:55.057654 28868 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:24:55.062101 28868 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data/instance:
uuid: "f1e4550f9a844a8f8d0a83fde9d0c756"
format_stamp: "Formatted at 2025-06-26 08:24:55 on dist-test-slave-z1j3"
I20250626 08:24:55.063292 28868 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/wal/instance:
uuid: "f1e4550f9a844a8f8d0a83fde9d0c756"
format_stamp: "Formatted at 2025-06-26 08:24:55 on dist-test-slave-z1j3"
I20250626 08:24:55.070410 28868 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20250626 08:24:55.076038 28884 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:55.077059 28868 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250626 08:24:55.077401 28868 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/wal
uuid: "f1e4550f9a844a8f8d0a83fde9d0c756"
format_stamp: "Formatted at 2025-06-26 08:24:55 on dist-test-slave-z1j3"
I20250626 08:24:55.077728 28868 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:24:55.129215 28868 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:24:55.130719 28868 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:24:55.131142 28868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:24:55.133637 28868 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:24:55.137708 28868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:24:55.137926 28868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:24:55.138183 28868 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:24:55.138402 28868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250626 08:24:55.280628 28868 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.67:37041
I20250626 08:24:55.280743 28996 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.67:37041 every 8 connection(s)
I20250626 08:24:55.283385 28868 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data/info.pb
I20250626 08:24:55.289438 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 28868
I20250626 08:24:55.290078 22785 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/wal/instance
I20250626 08:24:55.304695 28997 heartbeater.cc:344] Connected to a master server at 127.22.64.126:36125
I20250626 08:24:55.305148 28997 heartbeater.cc:461] Registering TS with master...
I20250626 08:24:55.306149 28997 heartbeater.cc:507] Master 127.22.64.126:36125 requested a full tablet report, sending...
I20250626 08:24:55.308480 28543 ts_manager.cc:194] Registered new tserver with Master: f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67:37041)
I20250626 08:24:55.309849 28543 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.67:59937
I20250626 08:24:55.310761 22785 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 08:24:55.340654 22785 test_util.cc:276] Using random seed: 1272747606
I20250626 08:24:55.380851 28543 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57144:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 08:24:55.383674 28543 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 08:24:55.440815 28666 tablet_service.cc:1468] Processing CreateTablet for tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 (DEFAULT_TABLE table=pre_rebuild [id=2de8aac6d31b440a9c92dcee358e0f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:24:55.440868 28932 tablet_service.cc:1468] Processing CreateTablet for tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 (DEFAULT_TABLE table=pre_rebuild [id=2de8aac6d31b440a9c92dcee358e0f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:24:55.443293 28666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:24:55.444011 28932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:24:55.442059 28799 tablet_service.cc:1468] Processing CreateTablet for tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 (DEFAULT_TABLE table=pre_rebuild [id=2de8aac6d31b440a9c92dcee358e0f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:24:55.447232 28799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:24:55.477825 29021 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Bootstrap starting.
I20250626 08:24:55.478096 29022 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: Bootstrap starting.
I20250626 08:24:55.484788 29022 tablet_bootstrap.cc:654] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: Neither blocks nor log segments found. Creating new log.
I20250626 08:24:55.487193 29021 tablet_bootstrap.cc:654] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Neither blocks nor log segments found. Creating new log.
I20250626 08:24:55.488034 29023 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: Bootstrap starting.
I20250626 08:24:55.488128 29022 log.cc:826] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: Log is configured to *not* fsync() on all Append() calls
I20250626 08:24:55.489813 29021 log.cc:826] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Log is configured to *not* fsync() on all Append() calls
I20250626 08:24:55.495800 29022 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: No bootstrap required, opened a new log
I20250626 08:24:55.496145 29021 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: No bootstrap required, opened a new log
I20250626 08:24:55.496246 29022 ts_tablet_manager.cc:1397] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.007s
I20250626 08:24:55.496546 29023 tablet_bootstrap.cc:654] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: Neither blocks nor log segments found. Creating new log.
I20250626 08:24:55.496750 29021 ts_tablet_manager.cc:1397] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.010s
I20250626 08:24:55.499631 29023 log.cc:826] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: Log is configured to *not* fsync() on all Append() calls
I20250626 08:24:55.505473 29023 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: No bootstrap required, opened a new log
I20250626 08:24:55.506006 29023 ts_tablet_manager.cc:1397] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.005s
I20250626 08:24:55.518129 29022 raft_consensus.cc:357] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:24:55.519133 29022 raft_consensus.cc:383] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:24:55.519413 29022 raft_consensus.cc:738] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e4550f9a844a8f8d0a83fde9d0c756, State: Initialized, Role: FOLLOWER
I20250626 08:24:55.520184 29022 consensus_queue.cc:260] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:24:55.523422 28997 heartbeater.cc:499] Master 127.22.64.126:36125 was elected leader, sending a full tablet report...
I20250626 08:24:55.524791 29022 ts_tablet_manager.cc:1428] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: Time spent starting tablet: real 0.028s user 0.019s sys 0.007s
I20250626 08:24:55.524892 29021 raft_consensus.cc:357] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:24:55.525842 29021 raft_consensus.cc:383] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:24:55.526144 29021 raft_consensus.cc:738] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d564fdd08441bb9df6532ecb8cd609, State: Initialized, Role: FOLLOWER
I20250626 08:24:55.527096 29021 consensus_queue.cc:260] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:24:55.531458 29021 ts_tablet_manager.cc:1428] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Time spent starting tablet: real 0.034s user 0.033s sys 0.000s
I20250626 08:24:55.534279 29023 raft_consensus.cc:357] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:24:55.535274 29023 raft_consensus.cc:383] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:24:55.535615 29023 raft_consensus.cc:738] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d56dd9e3c54186a39d8a9a59f8cc9c, State: Initialized, Role: FOLLOWER
I20250626 08:24:55.536535 29023 consensus_queue.cc:260] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:24:55.540424 29023 ts_tablet_manager.cc:1428] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: Time spent starting tablet: real 0.034s user 0.032s sys 0.000s
W20250626 08:24:55.541451 28998 tablet.cc:2378] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 08:24:55.601939 28732 tablet.cc:2378] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:24:55.628175 29028 raft_consensus.cc:491] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:55.628711 29028 raft_consensus.cc:513] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:24:55.631122 29028 leader_election.cc:290] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55d56dd9e3c54186a39d8a9a59f8cc9c (127.22.64.66:36331), f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67:37041)
I20250626 08:24:55.643290 28952 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfc809bd1f7f40fe8d2d5a8ebcb5fe68" candidate_uuid: "11d564fdd08441bb9df6532ecb8cd609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" is_pre_election: true
I20250626 08:24:55.643273 28819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfc809bd1f7f40fe8d2d5a8ebcb5fe68" candidate_uuid: "11d564fdd08441bb9df6532ecb8cd609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" is_pre_election: true
I20250626 08:24:55.644162 28952 raft_consensus.cc:2466] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11d564fdd08441bb9df6532ecb8cd609 in term 0.
I20250626 08:24:55.644167 28819 raft_consensus.cc:2466] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11d564fdd08441bb9df6532ecb8cd609 in term 0.
I20250626 08:24:55.645473 28619 leader_election.cc:304] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d564fdd08441bb9df6532ecb8cd609, f1e4550f9a844a8f8d0a83fde9d0c756; no voters:
I20250626 08:24:55.646183 29028 raft_consensus.cc:2802] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:24:55.646560 29028 raft_consensus.cc:491] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:55.646782 29028 raft_consensus.cc:3058] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:24:55.651238 29028 raft_consensus.cc:513] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:24:55.652649 29028 leader_election.cc:290] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [CANDIDATE]: Term 1 election: Requested vote from peers 55d56dd9e3c54186a39d8a9a59f8cc9c (127.22.64.66:36331), f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67:37041)
I20250626 08:24:55.653666 28819 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfc809bd1f7f40fe8d2d5a8ebcb5fe68" candidate_uuid: "11d564fdd08441bb9df6532ecb8cd609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c"
I20250626 08:24:55.653862 28952 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfc809bd1f7f40fe8d2d5a8ebcb5fe68" candidate_uuid: "11d564fdd08441bb9df6532ecb8cd609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756"
I20250626 08:24:55.654256 28819 raft_consensus.cc:3058] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:24:55.654417 28952 raft_consensus.cc:3058] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:24:55.661443 28819 raft_consensus.cc:2466] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11d564fdd08441bb9df6532ecb8cd609 in term 1.
I20250626 08:24:55.661443 28952 raft_consensus.cc:2466] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11d564fdd08441bb9df6532ecb8cd609 in term 1.
I20250626 08:24:55.662698 28622 leader_election.cc:304] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d564fdd08441bb9df6532ecb8cd609, 55d56dd9e3c54186a39d8a9a59f8cc9c; no voters:
I20250626 08:24:55.663409 29028 raft_consensus.cc:2802] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:24:55.664952 29028 raft_consensus.cc:695] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 1 LEADER]: Becoming Leader. State: Replica: 11d564fdd08441bb9df6532ecb8cd609, State: Running, Role: LEADER
I20250626 08:24:55.665688 29028 consensus_queue.cc:237] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:24:55.676347 28542 catalog_manager.cc:5582] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11d564fdd08441bb9df6532ecb8cd609 (127.22.64.65). New cstate: current_term: 1 leader_uuid: "11d564fdd08441bb9df6532ecb8cd609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } health_report { overall_health: HEALTHY } } }
W20250626 08:24:55.703683 28865 tablet.cc:2378] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:24:55.877564 28952 raft_consensus.cc:1273] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756 [term 1 FOLLOWER]: Refusing update from remote peer 11d564fdd08441bb9df6532ecb8cd609: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:24:55.879683 29028 consensus_queue.cc:1035] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:24:55.880757 28819 raft_consensus.cc:1273] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 1 FOLLOWER]: Refusing update from remote peer 11d564fdd08441bb9df6532ecb8cd609: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:24:55.882416 29028 consensus_queue.cc:1035] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:24:55.952525 29041 mvcc.cc:204] Tried to move back new op lower bound from 7171794107929276416 to 7171794107054145536. Current Snapshot: MvccSnapshot[applied={T|T < 7171794107895205888}]
I20250626 08:25:01.118551 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 28509
W20250626 08:25:01.530572 29076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:25:01.531435 29076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:25:01.569495 29076 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 08:25:02.005069 28864 heartbeater.cc:646] Failed to heartbeat to 127.22.64.126:36125 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.64.126:36125: connect: Connection refused (error 111)
W20250626 08:25:02.043201 28731 heartbeater.cc:646] Failed to heartbeat to 127.22.64.126:36125 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.64.126:36125: connect: Connection refused (error 111)
W20250626 08:25:02.065950 28997 heartbeater.cc:646] Failed to heartbeat to 127.22.64.126:36125 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.64.126:36125: connect: Connection refused (error 111)
W20250626 08:25:03.020105 29081 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 29076
W20250626 08:25:03.020622 29081 kernel_stack_watchdog.cc:198] Thread 29076 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250626 08:25:03.153642 29076 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.536s user 0.001s sys 0.000s
W20250626 08:25:03.254710 29076 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.638s user 0.004s sys 0.012s
I20250626 08:25:03.337210 29076 minidump.cc:252] Setting minidump size limit to 20M
I20250626 08:25:03.339179 29076 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:25:03.340317 29076 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:25:03.350932 29110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:25:03.351955 29111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:25:03.354507 29076 server_base.cc:1048] running on GCE node
W20250626 08:25:03.354804 29113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:25:03.355937 29076 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250626 08:25:03.356359 29076 hybrid_clock.cc:648] HybridClock initialized: now 1750926303356332 us; error 29533 us; skew 500 ppm
I20250626 08:25:03.357019 29076 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:25:03.361317 29076 webserver.cc:469] Webserver started at http://0.0.0.0:33799/ using document root <none> and password file <none>
I20250626 08:25:03.362134 29076 fs_manager.cc:362] Metadata directory not provided
I20250626 08:25:03.362399 29076 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:25:03.362835 29076 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:25:03.367156 29076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data/instance:
uuid: "75a4b7bca406441d9226163536095fbf"
format_stamp: "Formatted at 2025-06-26 08:25:03 on dist-test-slave-z1j3"
I20250626 08:25:03.368256 29076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal/instance:
uuid: "75a4b7bca406441d9226163536095fbf"
format_stamp: "Formatted at 2025-06-26 08:25:03 on dist-test-slave-z1j3"
I20250626 08:25:03.374419 29076 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250626 08:25:03.379179 29118 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:25:03.380118 29076 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250626 08:25:03.380455 29076 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal
uuid: "75a4b7bca406441d9226163536095fbf"
format_stamp: "Formatted at 2025-06-26 08:25:03 on dist-test-slave-z1j3"
I20250626 08:25:03.380782 29076 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:25:03.432226 29076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:25:03.433696 29076 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:25:03.434139 29076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:25:03.438927 29076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:25:03.454622 29076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf: Bootstrap starting.
I20250626 08:25:03.459873 29076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf: Neither blocks nor log segments found. Creating new log.
I20250626 08:25:03.461619 29076 log.cc:826] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf: Log is configured to *not* fsync() on all Append() calls
I20250626 08:25:03.466321 29076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf: No bootstrap required, opened a new log
I20250626 08:25:03.483001 29076 raft_consensus.cc:357] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a4b7bca406441d9226163536095fbf" member_type: VOTER }
I20250626 08:25:03.483660 29076 raft_consensus.cc:383] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:25:03.483901 29076 raft_consensus.cc:738] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a4b7bca406441d9226163536095fbf, State: Initialized, Role: FOLLOWER
I20250626 08:25:03.484640 29076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a4b7bca406441d9226163536095fbf" member_type: VOTER }
I20250626 08:25:03.485199 29076 raft_consensus.cc:397] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:25:03.485512 29076 raft_consensus.cc:491] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:25:03.485813 29076 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:25:03.490018 29076 raft_consensus.cc:513] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a4b7bca406441d9226163536095fbf" member_type: VOTER }
I20250626 08:25:03.490782 29076 leader_election.cc:304] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75a4b7bca406441d9226163536095fbf; no voters:
I20250626 08:25:03.492568 29076 leader_election.cc:290] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:25:03.492810 29125 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:25:03.496315 29125 raft_consensus.cc:695] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 1 LEADER]: Becoming Leader. State: Replica: 75a4b7bca406441d9226163536095fbf, State: Running, Role: LEADER
I20250626 08:25:03.497092 29125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a4b7bca406441d9226163536095fbf" member_type: VOTER }
I20250626 08:25:03.504009 29127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75a4b7bca406441d9226163536095fbf. Latest consensus state: current_term: 1 leader_uuid: "75a4b7bca406441d9226163536095fbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a4b7bca406441d9226163536095fbf" member_type: VOTER } }
I20250626 08:25:03.504323 29126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75a4b7bca406441d9226163536095fbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a4b7bca406441d9226163536095fbf" member_type: VOTER } }
I20250626 08:25:03.504915 29126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [sys.catalog]: This master's current role is: LEADER
I20250626 08:25:03.505400 29127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [sys.catalog]: This master's current role is: LEADER
I20250626 08:25:03.517746 29076 tablet_replica.cc:331] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf: stopping tablet replica
I20250626 08:25:03.518393 29076 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 1 LEADER]: Raft consensus shutting down.
I20250626 08:25:03.518833 29076 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:25:03.520984 29076 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250626 08:25:03.521454 29076 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250626 08:25:03.579378 29076 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250626 08:25:04.611948 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 28602
I20250626 08:25:04.662609 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 28735
I20250626 08:25:04.699586 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 28868
I20250626 08:25:04.740072 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:36125
--webserver_interface=127.22.64.126
--webserver_port=37167
--builtin_ntp_servers=127.22.64.84:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.64.126:36125 with env {}
W20250626 08:25:05.051846 29135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:25:05.052484 29135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:25:05.052976 29135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:25:05.085026 29135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 08:25:05.085395 29135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:25:05.085655 29135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 08:25:05.085894 29135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 08:25:05.121273 29135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.64.126:36125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.64.126:36125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.64.126
--webserver_port=37167
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:25:05.122725 29135 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:25:05.124475 29135 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:25:05.139362 29144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:25:05.139389 29142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:25:05.139977 29141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:25:06.322072 29143 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:25:06.322165 29135 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:25:06.326714 29135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:25:06.330009 29135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:25:06.331525 29135 hybrid_clock.cc:648] HybridClock initialized: now 1750926306331483 us; error 53 us; skew 500 ppm
I20250626 08:25:06.332356 29135 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:25:06.339234 29135 webserver.cc:469] Webserver started at http://127.22.64.126:37167/ using document root <none> and password file <none>
I20250626 08:25:06.340166 29135 fs_manager.cc:362] Metadata directory not provided
I20250626 08:25:06.340382 29135 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:25:06.348165 29135 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20250626 08:25:06.352703 29151 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:25:06.353821 29135 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250626 08:25:06.354131 29135 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal
uuid: "75a4b7bca406441d9226163536095fbf"
format_stamp: "Formatted at 2025-06-26 08:25:03 on dist-test-slave-z1j3"
I20250626 08:25:06.356069 29135 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:25:06.418298 29135 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:25:06.419836 29135 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:25:06.420359 29135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:25:06.493105 29135 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.126:36125
I20250626 08:25:06.493187 29202 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.126:36125 every 8 connection(s)
I20250626 08:25:06.495996 29135 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data/info.pb
I20250626 08:25:06.502717 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 29135
I20250626 08:25:06.504987 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.65:44937
--local_ip_for_outbound_sockets=127.22.64.65
--tserver_master_addrs=127.22.64.126:36125
--webserver_port=39355
--webserver_interface=127.22.64.65
--builtin_ntp_servers=127.22.64.84:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 08:25:06.508600 29203 sys_catalog.cc:263] Verifying existing consensus state
I20250626 08:25:06.525729 29203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf: Bootstrap starting.
I20250626 08:25:06.536470 29203 log.cc:826] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf: Log is configured to *not* fsync() on all Append() calls
I20250626 08:25:06.549285 29203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:25:06.550159 29203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf: Bootstrap complete.
I20250626 08:25:06.572861 29203 raft_consensus.cc:357] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a4b7bca406441d9226163536095fbf" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 36125 } }
I20250626 08:25:06.573534 29203 raft_consensus.cc:738] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a4b7bca406441d9226163536095fbf, State: Initialized, Role: FOLLOWER
I20250626 08:25:06.574360 29203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a4b7bca406441d9226163536095fbf" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 36125 } }
I20250626 08:25:06.574862 29203 raft_consensus.cc:397] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:25:06.575146 29203 raft_consensus.cc:491] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:25:06.575457 29203 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:25:06.579788 29203 raft_consensus.cc:513] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a4b7bca406441d9226163536095fbf" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 36125 } }
I20250626 08:25:06.580596 29203 leader_election.cc:304] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75a4b7bca406441d9226163536095fbf; no voters:
I20250626 08:25:06.582906 29203 leader_election.cc:290] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 08:25:06.583240 29207 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:25:06.586685 29207 raft_consensus.cc:695] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [term 2 LEADER]: Becoming Leader. State: Replica: 75a4b7bca406441d9226163536095fbf, State: Running, Role: LEADER
I20250626 08:25:06.587622 29207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a4b7bca406441d9226163536095fbf" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 36125 } }
I20250626 08:25:06.588390 29203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:25:06.597918 29208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "75a4b7bca406441d9226163536095fbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a4b7bca406441d9226163536095fbf" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 36125 } } }
I20250626 08:25:06.599303 29209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75a4b7bca406441d9226163536095fbf. Latest consensus state: current_term: 2 leader_uuid: "75a4b7bca406441d9226163536095fbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a4b7bca406441d9226163536095fbf" member_type: VOTER last_known_addr { host: "127.22.64.126" port: 36125 } } }
I20250626 08:25:06.600049 29209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [sys.catalog]: This master's current role is: LEADER
I20250626 08:25:06.603497 29208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf [sys.catalog]: This master's current role is: LEADER
I20250626 08:25:06.612275 29213 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:25:06.625775 29213 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=c76cf853180e4e76a6e0c4250c8b4545]
I20250626 08:25:06.634361 29213 tablet_loader.cc:96] loaded metadata for tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 (table pre_rebuild [id=c76cf853180e4e76a6e0c4250c8b4545])
I20250626 08:25:06.636108 29213 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:25:06.658422 29213 catalog_manager.cc:1349] Generated new cluster ID: a965e08b6c37454a88f25676876de570
I20250626 08:25:06.658829 29213 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:25:06.708498 29213 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 08:25:06.710865 29213 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:25:06.729467 29213 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf: Generated new TSK 0
I20250626 08:25:06.730839 29213 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250626 08:25:06.897754 29205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:25:06.898383 29205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:25:06.898922 29205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:25:06.931103 29205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:25:06.931995 29205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.65
I20250626 08:25:06.967941 29205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.65:44937
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.64.65
--webserver_port=39355
--tserver_master_addrs=127.22.64.126:36125
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.65
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:25:06.969385 29205 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:25:06.971148 29205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:25:06.989272 29232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:25:06.992919 29205 server_base.cc:1048] running on GCE node
W20250626 08:25:06.992189 29234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:25:06.992413 29231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:25:08.180691 29205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:25:08.183668 29205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:25:08.185135 29205 hybrid_clock.cc:648] HybridClock initialized: now 1750926308185074 us; error 78 us; skew 500 ppm
I20250626 08:25:08.185984 29205 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:25:08.199470 29205 webserver.cc:469] Webserver started at http://127.22.64.65:39355/ using document root <none> and password file <none>
I20250626 08:25:08.200526 29205 fs_manager.cc:362] Metadata directory not provided
I20250626 08:25:08.200759 29205 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:25:08.209441 29205 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20250626 08:25:08.214835 29242 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:25:08.216018 29205 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250626 08:25:08.216348 29205 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/wal
uuid: "11d564fdd08441bb9df6532ecb8cd609"
format_stamp: "Formatted at 2025-06-26 08:24:51 on dist-test-slave-z1j3"
I20250626 08:25:08.218410 29205 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:25:08.273588 29205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:25:08.275244 29205 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:25:08.275693 29205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:25:08.278810 29205 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:25:08.285176 29249 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 08:25:08.292956 29205 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 08:25:08.293196 29205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.001s
I20250626 08:25:08.293505 29205 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 08:25:08.298278 29205 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 08:25:08.298542 29205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250626 08:25:08.298940 29249 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Bootstrap starting.
I20250626 08:25:08.516327 29205 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.65:44937
I20250626 08:25:08.516534 29355 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.65:44937 every 8 connection(s)
I20250626 08:25:08.519372 29205 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data/info.pb
I20250626 08:25:08.527514 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 29205
I20250626 08:25:08.530339 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.66:36331
--local_ip_for_outbound_sockets=127.22.64.66
--tserver_master_addrs=127.22.64.126:36125
--webserver_port=44189
--webserver_interface=127.22.64.66
--builtin_ntp_servers=127.22.64.84:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 08:25:08.576293 29356 heartbeater.cc:344] Connected to a master server at 127.22.64.126:36125
I20250626 08:25:08.576829 29356 heartbeater.cc:461] Registering TS with master...
I20250626 08:25:08.578115 29356 heartbeater.cc:507] Master 127.22.64.126:36125 requested a full tablet report, sending...
I20250626 08:25:08.584528 29168 ts_manager.cc:194] Registered new tserver with Master: 11d564fdd08441bb9df6532ecb8cd609 (127.22.64.65:44937)
I20250626 08:25:08.591648 29168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.65:38929
I20250626 08:25:08.605989 29249 log.cc:826] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Log is configured to *not* fsync() on all Append() calls
W20250626 08:25:08.867643 29360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:25:08.868173 29360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:25:08.868702 29360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:25:08.901348 29360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:25:08.902248 29360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.66
I20250626 08:25:08.937935 29360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.66:36331
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.64.66
--webserver_port=44189
--tserver_master_addrs=127.22.64.126:36125
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.66
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:25:08.939344 29360 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:25:08.941040 29360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:25:08.958905 29368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:25:09.596237 29356 heartbeater.cc:499] Master 127.22.64.126:36125 was elected leader, sending a full tablet report...
W20250626 08:25:08.958915 29370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:25:08.960011 29367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:25:10.147068 29369 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:25:10.147393 29360 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:25:10.151705 29360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:25:10.158577 29360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:25:10.160104 29360 hybrid_clock.cc:648] HybridClock initialized: now 1750926310160036 us; error 86 us; skew 500 ppm
I20250626 08:25:10.160950 29360 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:25:10.168588 29360 webserver.cc:469] Webserver started at http://127.22.64.66:44189/ using document root <none> and password file <none>
I20250626 08:25:10.169551 29360 fs_manager.cc:362] Metadata directory not provided
I20250626 08:25:10.169793 29360 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:25:10.178239 29360 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20250626 08:25:10.183526 29377 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:25:10.184670 29360 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250626 08:25:10.184999 29360 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/wal
uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c"
format_stamp: "Formatted at 2025-06-26 08:24:53 on dist-test-slave-z1j3"
I20250626 08:25:10.187037 29360 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:25:10.243094 29360 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:25:10.244663 29360 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:25:10.245116 29360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:25:10.248378 29360 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:25:10.255702 29384 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 08:25:10.263542 29360 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 08:25:10.263780 29360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.003s
I20250626 08:25:10.264067 29360 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 08:25:10.268875 29360 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 08:25:10.269155 29360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.003s
I20250626 08:25:10.269605 29384 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: Bootstrap starting.
I20250626 08:25:10.453524 29360 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.66:36331
I20250626 08:25:10.453701 29490 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.66:36331 every 8 connection(s)
I20250626 08:25:10.457322 29360 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data/info.pb
I20250626 08:25:10.458688 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 29360
I20250626 08:25:10.460570 22785 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
/tmp/dist-test-taskbnla2f/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.64.67:37041
--local_ip_for_outbound_sockets=127.22.64.67
--tserver_master_addrs=127.22.64.126:36125
--webserver_port=44025
--webserver_interface=127.22.64.67
--builtin_ntp_servers=127.22.64.84:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 08:25:10.527505 29491 heartbeater.cc:344] Connected to a master server at 127.22.64.126:36125
I20250626 08:25:10.528034 29491 heartbeater.cc:461] Registering TS with master...
I20250626 08:25:10.529321 29491 heartbeater.cc:507] Master 127.22.64.126:36125 requested a full tablet report, sending...
I20250626 08:25:10.533504 29168 ts_manager.cc:194] Registered new tserver with Master: 55d56dd9e3c54186a39d8a9a59f8cc9c (127.22.64.66:36331)
I20250626 08:25:10.536803 29168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.66:51497
I20250626 08:25:10.629524 29384 log.cc:826] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: Log is configured to *not* fsync() on all Append() calls
I20250626 08:25:10.705662 29249 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:25:10.706590 29249 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Bootstrap complete.
I20250626 08:25:10.707989 29249 ts_tablet_manager.cc:1397] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Time spent bootstrapping tablet: real 2.409s user 2.321s sys 0.053s
I20250626 08:25:10.719419 29249 raft_consensus.cc:357] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:10.721606 29249 raft_consensus.cc:738] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d564fdd08441bb9df6532ecb8cd609, State: Initialized, Role: FOLLOWER
I20250626 08:25:10.722519 29249 consensus_queue.cc:260] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:10.726890 29249 ts_tablet_manager.cc:1428] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
W20250626 08:25:10.908428 29492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 08:25:10.908951 29492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 08:25:10.909468 29492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 08:25:10.941391 29492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 08:25:10.942507 29492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.64.67
I20250626 08:25:10.978165 29492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.64.84:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.64.67:37041
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.64.67
--webserver_port=44025
--tserver_master_addrs=127.22.64.126:36125
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.64.67
--log_dir=/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 08:06:47 UTC on 24a791456cd2
build id 6777
TSAN enabled
I20250626 08:25:10.979552 29492 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:25:10.981248 29492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:25:10.998627 29504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:25:11.541483 29491 heartbeater.cc:499] Master 127.22.64.126:36125 was elected leader, sending a full tablet report...
I20250626 08:25:12.127161 29509 raft_consensus.cc:491] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:25:12.128196 29509 raft_consensus.cc:513] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:12.146201 29509 leader_election.cc:290] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55d56dd9e3c54186a39d8a9a59f8cc9c (127.22.64.66:36331), f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67:37041)
W20250626 08:25:12.159221 29243 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.64.67:37041: connect: Connection refused (error 111)
W20250626 08:25:12.185755 29243 leader_election.cc:336] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67:37041): Network error: Client connection negotiation failed: client connection to 127.22.64.67:37041: connect: Connection refused (error 111)
I20250626 08:25:12.164467 29446 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfc809bd1f7f40fe8d2d5a8ebcb5fe68" candidate_uuid: "11d564fdd08441bb9df6532ecb8cd609" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" is_pre_election: true
W20250626 08:25:12.189685 29246 leader_election.cc:343] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55d56dd9e3c54186a39d8a9a59f8cc9c (127.22.64.66:36331): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:25:12.190413 29246 leader_election.cc:304] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11d564fdd08441bb9df6532ecb8cd609; no voters: 55d56dd9e3c54186a39d8a9a59f8cc9c, f1e4550f9a844a8f8d0a83fde9d0c756
I20250626 08:25:12.192505 29509 raft_consensus.cc:2747] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250626 08:25:10.998710 29506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:25:11.001377 29503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:25:12.533879 29505 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1535 milliseconds
I20250626 08:25:12.533912 29492 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 08:25:12.539539 29492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 08:25:12.541908 29492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 08:25:12.543264 29492 hybrid_clock.cc:648] HybridClock initialized: now 1750926312543234 us; error 55 us; skew 500 ppm
I20250626 08:25:12.544064 29492 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:25:12.550681 29492 webserver.cc:469] Webserver started at http://127.22.64.67:44025/ using document root <none> and password file <none>
I20250626 08:25:12.551805 29492 fs_manager.cc:362] Metadata directory not provided
I20250626 08:25:12.552042 29492 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:25:12.560875 29492 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20250626 08:25:12.566152 29518 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:25:12.567281 29492 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 08:25:12.567592 29492 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/wal
uuid: "f1e4550f9a844a8f8d0a83fde9d0c756"
format_stamp: "Formatted at 2025-06-26 08:24:55 on dist-test-slave-z1j3"
I20250626 08:25:12.569519 29492 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:25:12.656834 29492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:25:12.658455 29492 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:25:12.658897 29492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:25:12.661499 29492 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:25:12.668051 29525 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 08:25:12.678987 29492 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 08:25:12.679207 29492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250626 08:25:12.679513 29492 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 08:25:12.684194 29492 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 08:25:12.684393 29492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250626 08:25:12.684870 29525 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: Bootstrap starting.
I20250626 08:25:12.872006 29492 rpc_server.cc:307] RPC server started. Bound to: 127.22.64.67:37041
I20250626 08:25:12.872210 29631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.64.67:37041 every 8 connection(s)
I20250626 08:25:12.875772 29492 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data/info.pb
I20250626 08:25:12.883003 22785 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu as pid 29492
I20250626 08:25:12.915879 29632 heartbeater.cc:344] Connected to a master server at 127.22.64.126:36125
I20250626 08:25:12.916394 29632 heartbeater.cc:461] Registering TS with master...
I20250626 08:25:12.917654 29632 heartbeater.cc:507] Master 127.22.64.126:36125 requested a full tablet report, sending...
I20250626 08:25:12.921557 29168 ts_manager.cc:194] Registered new tserver with Master: f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67:37041)
I20250626 08:25:12.924734 29168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.64.67:60395
I20250626 08:25:12.934499 22785 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 08:25:12.995857 29525 log.cc:826] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: Log is configured to *not* fsync() on all Append() calls
I20250626 08:25:13.550571 29384 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:25:13.551401 29384 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: Bootstrap complete.
I20250626 08:25:13.552942 29384 ts_tablet_manager.cc:1397] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: Time spent bootstrapping tablet: real 3.284s user 3.209s sys 0.057s
I20250626 08:25:13.559465 29384 raft_consensus.cc:357] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:13.561415 29384 raft_consensus.cc:738] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d56dd9e3c54186a39d8a9a59f8cc9c, State: Initialized, Role: FOLLOWER
I20250626 08:25:13.562146 29384 consensus_queue.cc:260] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:13.565397 29384 ts_tablet_manager.cc:1428] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20250626 08:25:13.928592 29632 heartbeater.cc:499] Master 127.22.64.126:36125 was elected leader, sending a full tablet report...
I20250626 08:25:14.257305 29645 raft_consensus.cc:491] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:25:14.257784 29645 raft_consensus.cc:513] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:14.259310 29645 leader_election.cc:290] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55d56dd9e3c54186a39d8a9a59f8cc9c (127.22.64.66:36331), f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67:37041)
I20250626 08:25:14.260337 29446 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfc809bd1f7f40fe8d2d5a8ebcb5fe68" candidate_uuid: "11d564fdd08441bb9df6532ecb8cd609" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" is_pre_election: true
I20250626 08:25:14.261056 29446 raft_consensus.cc:2466] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11d564fdd08441bb9df6532ecb8cd609 in term 1.
I20250626 08:25:14.262434 29246 leader_election.cc:304] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d564fdd08441bb9df6532ecb8cd609, 55d56dd9e3c54186a39d8a9a59f8cc9c; no voters:
I20250626 08:25:14.263918 29645 raft_consensus.cc:2802] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:25:14.264438 29645 raft_consensus.cc:491] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:25:14.264897 29645 raft_consensus.cc:3058] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:25:14.272588 29645 raft_consensus.cc:513] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:14.274536 29645 leader_election.cc:290] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [CANDIDATE]: Term 2 election: Requested vote from peers 55d56dd9e3c54186a39d8a9a59f8cc9c (127.22.64.66:36331), f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67:37041)
I20250626 08:25:14.275337 29446 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfc809bd1f7f40fe8d2d5a8ebcb5fe68" candidate_uuid: "11d564fdd08441bb9df6532ecb8cd609" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c"
I20250626 08:25:14.275907 29446 raft_consensus.cc:3058] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:25:14.276502 29587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfc809bd1f7f40fe8d2d5a8ebcb5fe68" candidate_uuid: "11d564fdd08441bb9df6532ecb8cd609" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" is_pre_election: true
I20250626 08:25:14.276877 29586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfc809bd1f7f40fe8d2d5a8ebcb5fe68" candidate_uuid: "11d564fdd08441bb9df6532ecb8cd609" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756"
I20250626 08:25:14.285189 29446 raft_consensus.cc:2466] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11d564fdd08441bb9df6532ecb8cd609 in term 2.
W20250626 08:25:14.285706 29243 leader_election.cc:343] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67:37041): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:25:14.286425 29246 leader_election.cc:304] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d564fdd08441bb9df6532ecb8cd609, 55d56dd9e3c54186a39d8a9a59f8cc9c; no voters:
I20250626 08:25:14.287197 29645 raft_consensus.cc:2802] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:25:14.288569 29645 raft_consensus.cc:695] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 2 LEADER]: Becoming Leader. State: Replica: 11d564fdd08441bb9df6532ecb8cd609, State: Running, Role: LEADER
I20250626 08:25:14.289302 29645 consensus_queue.cc:237] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:14.298542 29168 catalog_manager.cc:5582] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 reported cstate change: term changed from 0 to 2, leader changed from <none> to 11d564fdd08441bb9df6532ecb8cd609 (127.22.64.65), VOTER 11d564fdd08441bb9df6532ecb8cd609 (127.22.64.65) added, VOTER 55d56dd9e3c54186a39d8a9a59f8cc9c (127.22.64.66) added, VOTER f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67) added. New cstate: current_term: 2 leader_uuid: "11d564fdd08441bb9df6532ecb8cd609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } health_report { overall_health: HEALTHY } } }
W20250626 08:25:14.860674 22785 scanner-internal.cc:458] Time spent opening tablet: real 1.890s user 0.008s sys 0.004s
I20250626 08:25:14.902717 29446 raft_consensus.cc:1273] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 2 FOLLOWER]: Refusing update from remote peer 11d564fdd08441bb9df6532ecb8cd609: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250626 08:25:14.904548 29658 consensus_queue.cc:1035] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.001s
W20250626 08:25:14.923207 29243 consensus_peers.cc:487] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 -> Peer f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67:37041): Couldn't send request to peer f1e4550f9a844a8f8d0a83fde9d0c756. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250626 08:25:14.995910 29352 debug-util.cc:398] Leaking SignalData structure 0x7b08000c0b40 after lost signal to thread 29226
I20250626 08:25:15.091230 29311 consensus_queue.cc:237] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:15.097121 29446 raft_consensus.cc:1273] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 2 FOLLOWER]: Refusing update from remote peer 11d564fdd08441bb9df6532ecb8cd609: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250626 08:25:15.098883 29661 consensus_queue.cc:1035] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20250626 08:25:15.106002 29661 raft_consensus.cc:2953] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } } }
I20250626 08:25:15.108700 29446 raft_consensus.cc:2953] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } } }
I20250626 08:25:15.131287 29154 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:25:15.137600 29168 catalog_manager.cc:5582] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 reported cstate change: config changed from index -1 to 207, VOTER f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67) evicted. New cstate: current_term: 2 leader_uuid: "11d564fdd08441bb9df6532ecb8cd609" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } health_report { overall_health: HEALTHY } } }
I20250626 08:25:15.194742 29311 consensus_queue.cc:237] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:15.197414 29661 raft_consensus.cc:2953] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 55d56dd9e3c54186a39d8a9a59f8cc9c (127.22.64.66) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } } }
I20250626 08:25:15.210382 29154 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:25:15.215123 29168 catalog_manager.cc:5582] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 reported cstate change: config changed from index 207 to 208, VOTER 55d56dd9e3c54186a39d8a9a59f8cc9c (127.22.64.66) evicted. New cstate: current_term: 2 leader_uuid: "11d564fdd08441bb9df6532ecb8cd609" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } health_report { overall_health: HEALTHY } } }
I20250626 08:25:15.219017 29567 tablet_service.cc:1515] Processing DeleteTablet for tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 with delete_type TABLET_DATA_TOMBSTONED (TS f1e4550f9a844a8f8d0a83fde9d0c756 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:57418
W20250626 08:25:15.237088 29152 catalog_manager.cc:4908] TS f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67:37041): delete failed for tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 already in progress: opening tablet
I20250626 08:25:15.267731 29525 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:25:15.269003 29525 tablet_bootstrap.cc:492] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: Bootstrap complete.
I20250626 08:25:15.268944 29426 tablet_service.cc:1515] Processing DeleteTablet for tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 with delete_type TABLET_DATA_TOMBSTONED (TS 55d56dd9e3c54186a39d8a9a59f8cc9c not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:56040
I20250626 08:25:15.271077 29525 ts_tablet_manager.cc:1397] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: Time spent bootstrapping tablet: real 2.587s user 2.474s sys 0.068s
I20250626 08:25:15.280964 29525 raft_consensus.cc:357] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:15.284422 29525 raft_consensus.cc:738] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e4550f9a844a8f8d0a83fde9d0c756, State: Initialized, Role: FOLLOWER
I20250626 08:25:15.285706 29525 consensus_queue.cc:260] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:15.287477 29672 tablet_replica.cc:331] stopping tablet replica
I20250626 08:25:15.288780 29672 raft_consensus.cc:2241] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:25:15.295461 29525 ts_tablet_manager.cc:1428] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: Time spent starting tablet: real 0.024s user 0.017s sys 0.004s
I20250626 08:25:15.298738 29672 raft_consensus.cc:2270] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:25:15.304546 29567 tablet_service.cc:1515] Processing DeleteTablet for tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:57418
I20250626 08:25:15.305603 29670 tablet_replica.cc:331] stopping tablet replica
I20250626 08:25:15.307412 29670 raft_consensus.cc:2241] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:25:15.307986 29670 raft_consensus.cc:2270] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:25:15.326673 29672 ts_tablet_manager.cc:1905] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:25:15.343055 29672 ts_tablet_manager.cc:1918] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250626 08:25:15.343649 29672 log.cc:1199] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 55d56dd9e3c54186a39d8a9a59f8cc9c: Deleting WAL directory at /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/wal/wals/dfc809bd1f7f40fe8d2d5a8ebcb5fe68
I20250626 08:25:15.345762 29155 catalog_manager.cc:4928] TS 55d56dd9e3c54186a39d8a9a59f8cc9c (127.22.64.66:36331): tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 (table pre_rebuild [id=c76cf853180e4e76a6e0c4250c8b4545]) successfully deleted
I20250626 08:25:15.351518 29670 ts_tablet_manager.cc:1905] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:25:15.370386 29670 ts_tablet_manager.cc:1918] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250626 08:25:15.370978 29670 log.cc:1199] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P f1e4550f9a844a8f8d0a83fde9d0c756: Deleting WAL directory at /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/wal/wals/dfc809bd1f7f40fe8d2d5a8ebcb5fe68
I20250626 08:25:15.372836 29152 catalog_manager.cc:4928] TS f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67:37041): tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 (table pre_rebuild [id=c76cf853180e4e76a6e0c4250c8b4545]) successfully deleted
I20250626 08:25:15.793440 29291 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 08:25:15.811555 29426 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 08:25:15.834053 29567 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
75a4b7bca406441d9226163536095fbf | 127.22.64.126:36125 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.64.84:39889 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
11d564fdd08441bb9df6532ecb8cd609 | 127.22.64.65:44937 | HEALTHY | <none> | 1 | 0
55d56dd9e3c54186a39d8a9a59f8cc9c | 127.22.64.66:36331 | HEALTHY | <none> | 0 | 0
f1e4550f9a844a8f8d0a83fde9d0c756 | 127.22.64.67:37041 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.22.64.65 | experimental | 127.22.64.65:44937
local_ip_for_outbound_sockets | 127.22.64.66 | experimental | 127.22.64.66:36331
local_ip_for_outbound_sockets | 127.22.64.67 | experimental | 127.22.64.67:37041
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data/info.pb | hidden | 127.22.64.65:44937
server_dump_info_path | /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data/info.pb | hidden | 127.22.64.66:36331
server_dump_info_path | /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data/info.pb | hidden | 127.22.64.67:37041
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.64.84:39889 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250626 08:25:16.197067 22785 log_verifier.cc:126] Checking tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68
I20250626 08:25:16.535985 22785 log_verifier.cc:177] Verified matching terms for 208 ops in tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68
I20250626 08:25:16.539117 29167 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54850:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250626 08:25:16.539942 29167 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54850:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250626 08:25:16.574874 29167 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 75a4b7bca406441d9226163536095fbf: Sending DeleteTablet for 1 replicas of tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68
I20250626 08:25:16.577206 22785 test_util.cc:276] Using random seed: 1293984156
I20250626 08:25:16.577605 29291 tablet_service.cc:1515] Processing DeleteTablet for tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 08:25:16 UTC) from {username='slave'} at 127.0.0.1:34738
I20250626 08:25:16.591327 29703 tablet_replica.cc:331] stopping tablet replica
I20250626 08:25:16.592332 29703 raft_consensus.cc:2241] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 2 LEADER]: Raft consensus shutting down.
I20250626 08:25:16.593127 29703 raft_consensus.cc:2270] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:25:16.636653 29703 ts_tablet_manager.cc:1905] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 08:25:16.655411 29703 ts_tablet_manager.cc:1918] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250626 08:25:16.655992 29703 log.cc:1199] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Deleting WAL directory at /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/wal/wals/dfc809bd1f7f40fe8d2d5a8ebcb5fe68
I20250626 08:25:16.657138 29703 ts_tablet_manager.cc:1939] T dfc809bd1f7f40fe8d2d5a8ebcb5fe68 P 11d564fdd08441bb9df6532ecb8cd609: Deleting consensus metadata
I20250626 08:25:16.660586 29154 catalog_manager.cc:4928] TS 11d564fdd08441bb9df6532ecb8cd609 (127.22.64.65:44937): tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68 (table pre_rebuild [id=c76cf853180e4e76a6e0c4250c8b4545]) successfully deleted
I20250626 08:25:16.674394 29167 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54898:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 08:25:16.677909 29167 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 08:25:16.707444 29426 tablet_service.cc:1468] Processing CreateTablet for tablet 4530f1216e9d4721a4cc9b9b6741383d (DEFAULT_TABLE table=post_rebuild [id=e37b5fc9ad4f4e509a3af5905c04d9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:25:16.708935 29426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4530f1216e9d4721a4cc9b9b6741383d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:25:16.710525 29567 tablet_service.cc:1468] Processing CreateTablet for tablet 4530f1216e9d4721a4cc9b9b6741383d (DEFAULT_TABLE table=post_rebuild [id=e37b5fc9ad4f4e509a3af5905c04d9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:25:16.711861 29567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4530f1216e9d4721a4cc9b9b6741383d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:25:16.734838 29291 tablet_service.cc:1468] Processing CreateTablet for tablet 4530f1216e9d4721a4cc9b9b6741383d (DEFAULT_TABLE table=post_rebuild [id=e37b5fc9ad4f4e509a3af5905c04d9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 08:25:16.736339 29291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4530f1216e9d4721a4cc9b9b6741383d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:25:16.737143 29710 tablet_bootstrap.cc:492] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c: Bootstrap starting.
I20250626 08:25:16.743902 29710 tablet_bootstrap.cc:654] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c: Neither blocks nor log segments found. Creating new log.
I20250626 08:25:16.755707 29710 tablet_bootstrap.cc:492] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c: No bootstrap required, opened a new log
I20250626 08:25:16.756278 29710 ts_tablet_manager.cc:1397] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20250626 08:25:16.759438 29710 raft_consensus.cc:357] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:16.760414 29710 raft_consensus.cc:383] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:25:16.760782 29710 raft_consensus.cc:738] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d56dd9e3c54186a39d8a9a59f8cc9c, State: Initialized, Role: FOLLOWER
I20250626 08:25:16.761660 29710 consensus_queue.cc:260] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:16.778918 29710 ts_tablet_manager.cc:1428] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c: Time spent starting tablet: real 0.022s user 0.012s sys 0.007s
I20250626 08:25:16.780411 29713 tablet_bootstrap.cc:492] T 4530f1216e9d4721a4cc9b9b6741383d P 11d564fdd08441bb9df6532ecb8cd609: Bootstrap starting.
I20250626 08:25:16.781716 29714 tablet_bootstrap.cc:492] T 4530f1216e9d4721a4cc9b9b6741383d P f1e4550f9a844a8f8d0a83fde9d0c756: Bootstrap starting.
I20250626 08:25:16.790930 29712 raft_consensus.cc:491] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:25:16.791452 29712 raft_consensus.cc:513] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:16.793766 29714 tablet_bootstrap.cc:654] T 4530f1216e9d4721a4cc9b9b6741383d P f1e4550f9a844a8f8d0a83fde9d0c756: Neither blocks nor log segments found. Creating new log.
I20250626 08:25:16.796671 29712 leader_election.cc:290] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67:37041), 11d564fdd08441bb9df6532ecb8cd609 (127.22.64.65:44937)
I20250626 08:25:16.796860 29713 tablet_bootstrap.cc:654] T 4530f1216e9d4721a4cc9b9b6741383d P 11d564fdd08441bb9df6532ecb8cd609: Neither blocks nor log segments found. Creating new log.
I20250626 08:25:16.861447 29714 tablet_bootstrap.cc:492] T 4530f1216e9d4721a4cc9b9b6741383d P f1e4550f9a844a8f8d0a83fde9d0c756: No bootstrap required, opened a new log
I20250626 08:25:16.861919 29714 ts_tablet_manager.cc:1397] T 4530f1216e9d4721a4cc9b9b6741383d P f1e4550f9a844a8f8d0a83fde9d0c756: Time spent bootstrapping tablet: real 0.080s user 0.011s sys 0.024s
I20250626 08:25:16.864830 29714 raft_consensus.cc:357] T 4530f1216e9d4721a4cc9b9b6741383d P f1e4550f9a844a8f8d0a83fde9d0c756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:16.865638 29714 raft_consensus.cc:383] T 4530f1216e9d4721a4cc9b9b6741383d P f1e4550f9a844a8f8d0a83fde9d0c756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:25:16.866912 29713 tablet_bootstrap.cc:492] T 4530f1216e9d4721a4cc9b9b6741383d P 11d564fdd08441bb9df6532ecb8cd609: No bootstrap required, opened a new log
I20250626 08:25:16.867379 29713 ts_tablet_manager.cc:1397] T 4530f1216e9d4721a4cc9b9b6741383d P 11d564fdd08441bb9df6532ecb8cd609: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.030s
I20250626 08:25:16.868866 29311 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4530f1216e9d4721a4cc9b9b6741383d" candidate_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d564fdd08441bb9df6532ecb8cd609" is_pre_election: true
I20250626 08:25:16.870028 29713 raft_consensus.cc:357] T 4530f1216e9d4721a4cc9b9b6741383d P 11d564fdd08441bb9df6532ecb8cd609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:16.870723 29713 raft_consensus.cc:383] T 4530f1216e9d4721a4cc9b9b6741383d P 11d564fdd08441bb9df6532ecb8cd609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:25:16.871008 29713 raft_consensus.cc:738] T 4530f1216e9d4721a4cc9b9b6741383d P 11d564fdd08441bb9df6532ecb8cd609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d564fdd08441bb9df6532ecb8cd609, State: Initialized, Role: FOLLOWER
I20250626 08:25:16.871665 29713 consensus_queue.cc:260] T 4530f1216e9d4721a4cc9b9b6741383d P 11d564fdd08441bb9df6532ecb8cd609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:16.865942 29714 raft_consensus.cc:738] T 4530f1216e9d4721a4cc9b9b6741383d P f1e4550f9a844a8f8d0a83fde9d0c756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e4550f9a844a8f8d0a83fde9d0c756, State: Initialized, Role: FOLLOWER
W20250626 08:25:16.874765 29380 leader_election.cc:343] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 11d564fdd08441bb9df6532ecb8cd609 (127.22.64.65:44937): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:25:16.876124 29586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4530f1216e9d4721a4cc9b9b6741383d" candidate_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" is_pre_election: true
I20250626 08:25:16.875854 29714 consensus_queue.cc:260] T 4530f1216e9d4721a4cc9b9b6741383d P f1e4550f9a844a8f8d0a83fde9d0c756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:16.877703 29586 raft_consensus.cc:2466] T 4530f1216e9d4721a4cc9b9b6741383d P f1e4550f9a844a8f8d0a83fde9d0c756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55d56dd9e3c54186a39d8a9a59f8cc9c in term 0.
I20250626 08:25:16.891168 29378 leader_election.cc:304] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 55d56dd9e3c54186a39d8a9a59f8cc9c, f1e4550f9a844a8f8d0a83fde9d0c756; no voters: 11d564fdd08441bb9df6532ecb8cd609
I20250626 08:25:16.892107 29714 ts_tablet_manager.cc:1428] T 4530f1216e9d4721a4cc9b9b6741383d P f1e4550f9a844a8f8d0a83fde9d0c756: Time spent starting tablet: real 0.030s user 0.004s sys 0.005s
I20250626 08:25:16.892107 29712 raft_consensus.cc:2802] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:25:16.892485 29712 raft_consensus.cc:491] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:25:16.892810 29712 raft_consensus.cc:3058] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:25:16.895262 29633 tablet.cc:2378] T 4530f1216e9d4721a4cc9b9b6741383d P f1e4550f9a844a8f8d0a83fde9d0c756: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:25:16.899757 29712 raft_consensus.cc:513] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:16.902632 29586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4530f1216e9d4721a4cc9b9b6741383d" candidate_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756"
I20250626 08:25:16.903272 29586 raft_consensus.cc:3058] T 4530f1216e9d4721a4cc9b9b6741383d P f1e4550f9a844a8f8d0a83fde9d0c756 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:25:16.913612 29311 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4530f1216e9d4721a4cc9b9b6741383d" candidate_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d564fdd08441bb9df6532ecb8cd609"
I20250626 08:25:16.915266 29311 raft_consensus.cc:3058] T 4530f1216e9d4721a4cc9b9b6741383d P 11d564fdd08441bb9df6532ecb8cd609 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:25:16.916595 29712 leader_election.cc:290] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [CANDIDATE]: Term 1 election: Requested vote from peers f1e4550f9a844a8f8d0a83fde9d0c756 (127.22.64.67:37041), 11d564fdd08441bb9df6532ecb8cd609 (127.22.64.65:44937)
I20250626 08:25:16.922295 29586 raft_consensus.cc:2466] T 4530f1216e9d4721a4cc9b9b6741383d P f1e4550f9a844a8f8d0a83fde9d0c756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55d56dd9e3c54186a39d8a9a59f8cc9c in term 1.
I20250626 08:25:16.924993 29713 ts_tablet_manager.cc:1428] T 4530f1216e9d4721a4cc9b9b6741383d P 11d564fdd08441bb9df6532ecb8cd609: Time spent starting tablet: real 0.057s user 0.000s sys 0.018s
I20250626 08:25:16.929742 29311 raft_consensus.cc:2466] T 4530f1216e9d4721a4cc9b9b6741383d P 11d564fdd08441bb9df6532ecb8cd609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55d56dd9e3c54186a39d8a9a59f8cc9c in term 1.
I20250626 08:25:16.935133 29378 leader_election.cc:304] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55d56dd9e3c54186a39d8a9a59f8cc9c, f1e4550f9a844a8f8d0a83fde9d0c756; no voters:
I20250626 08:25:16.936017 29712 raft_consensus.cc:2802] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:25:16.943759 29712 raft_consensus.cc:695] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [term 1 LEADER]: Becoming Leader. State: Replica: 55d56dd9e3c54186a39d8a9a59f8cc9c, State: Running, Role: LEADER
I20250626 08:25:16.944746 29712 consensus_queue.cc:237] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } }
I20250626 08:25:16.956754 29167 catalog_manager.cc:5582] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 55d56dd9e3c54186a39d8a9a59f8cc9c (127.22.64.66). New cstate: current_term: 1 leader_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d56dd9e3c54186a39d8a9a59f8cc9c" member_type: VOTER last_known_addr { host: "127.22.64.66" port: 36331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 } health_report { overall_health: UNKNOWN } } }
W20250626 08:25:16.995201 29493 tablet.cc:2378] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 08:25:17.068125 29357 tablet.cc:2378] T 4530f1216e9d4721a4cc9b9b6741383d P 11d564fdd08441bb9df6532ecb8cd609: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:25:17.219614 29586 raft_consensus.cc:1273] T 4530f1216e9d4721a4cc9b9b6741383d P f1e4550f9a844a8f8d0a83fde9d0c756 [term 1 FOLLOWER]: Refusing update from remote peer 55d56dd9e3c54186a39d8a9a59f8cc9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:25:17.221742 29311 raft_consensus.cc:1273] T 4530f1216e9d4721a4cc9b9b6741383d P 11d564fdd08441bb9df6532ecb8cd609 [term 1 FOLLOWER]: Refusing update from remote peer 55d56dd9e3c54186a39d8a9a59f8cc9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:25:17.225486 29712 consensus_queue.cc:1035] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11d564fdd08441bb9df6532ecb8cd609" member_type: VOTER last_known_addr { host: "127.22.64.65" port: 44937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:25:17.227207 29712 consensus_queue.cc:1035] T 4530f1216e9d4721a4cc9b9b6741383d P 55d56dd9e3c54186a39d8a9a59f8cc9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e4550f9a844a8f8d0a83fde9d0c756" member_type: VOTER last_known_addr { host: "127.22.64.67" port: 37041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:25:17.274535 29733 mvcc.cc:204] Tried to move back new op lower bound from 7171794195307384832 to 7171794194216140800. Current Snapshot: MvccSnapshot[applied={T|T < 7171794195307384832}]
I20250626 08:25:17.289296 29734 mvcc.cc:204] Tried to move back new op lower bound from 7171794195351961600 to 7171794194216140800. Current Snapshot: MvccSnapshot[applied={T|T < 7171794195307384832}]
I20250626 08:25:17.299484 29735 mvcc.cc:204] Tried to move back new op lower bound from 7171794195307384832 to 7171794194216140800. Current Snapshot: MvccSnapshot[applied={T|T < 7171794195307384832}]
I20250626 08:25:22.670253 29426 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 08:25:22.671612 29291 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 08:25:22.672906 29567 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
75a4b7bca406441d9226163536095fbf | 127.22.64.126:36125 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.64.84:39889 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
11d564fdd08441bb9df6532ecb8cd609 | 127.22.64.65:44937 | HEALTHY | <none> | 0 | 0
55d56dd9e3c54186a39d8a9a59f8cc9c | 127.22.64.66:36331 | HEALTHY | <none> | 1 | 0
f1e4550f9a844a8f8d0a83fde9d0c756 | 127.22.64.67:37041 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.22.64.65 | experimental | 127.22.64.65:44937
local_ip_for_outbound_sockets | 127.22.64.66 | experimental | 127.22.64.66:36331
local_ip_for_outbound_sockets | 127.22.64.67 | experimental | 127.22.64.67:37041
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-0/data/info.pb | hidden | 127.22.64.65:44937
server_dump_info_path | /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-1/data/info.pb | hidden | 127.22.64.66:36331
server_dump_info_path | /tmp/dist-test-taskbnla2f/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750926137374372-22785-0/minicluster-data/ts-2/data/info.pb | hidden | 127.22.64.67:37041
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.22.64.84:39889 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250626 08:25:22.889183 22785 log_verifier.cc:126] Checking tablet 4530f1216e9d4721a4cc9b9b6741383d
I20250626 08:25:23.666263 22785 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 4530f1216e9d4721a4cc9b9b6741383d
I20250626 08:25:23.667302 22785 log_verifier.cc:126] Checking tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68
I20250626 08:25:23.667629 22785 log_verifier.cc:177] Verified matching terms for 0 ops in tablet dfc809bd1f7f40fe8d2d5a8ebcb5fe68
I20250626 08:25:23.694530 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 29205
I20250626 08:25:23.742182 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 29360
I20250626 08:25:23.791251 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 29492
I20250626 08:25:23.836426 22785 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbnla2f/build/tsan/bin/kudu with pid 29135
2025-06-26T08:25:23Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (36498 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (36498 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (186455 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST