Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250624 14:18:11.644501 2104 test_util.cc:276] Using random seed: -7093305
W20250624 14:18:12.863682 2104 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.173s user 0.418s sys 0.751s
W20250624 14:18:12.864161 2104 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.174s user 0.422s sys 0.751s
I20250624 14:18:12.866598 2104 ts_itest-base.cc:115] Starting cluster with:
I20250624 14:18:12.866868 2104 ts_itest-base.cc:116] --------------
I20250624 14:18:12.867074 2104 ts_itest-base.cc:117] 4 tablet servers
I20250624 14:18:12.867291 2104 ts_itest-base.cc:118] 3 replicas per TS
I20250624 14:18:12.867486 2104 ts_itest-base.cc:119] --------------
2025-06-24T14:18:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T14:18:12Z Disabled control of system clock
I20250624 14:18:12.914737 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:37963
--webserver_interface=127.2.14.62
--webserver_port=0
--builtin_ntp_servers=127.2.14.20:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.14.62:37963 with env {}
W20250624 14:18:13.217186 2118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:13.217800 2118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:18:13.218259 2118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:13.249668 2118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 14:18:13.250001 2118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:18:13.250254 2118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 14:18:13.250487 2118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 14:18:13.285943 2118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.14.62:37963
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:37963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:18:13.287297 2118 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:18:13.289018 2118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:18:13.305589 2124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:13.305619 2127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:13.305581 2125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:13.306871 2118 server_base.cc:1048] running on GCE node
I20250624 14:18:14.495095 2118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:18:14.498307 2118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:18:14.499816 2118 hybrid_clock.cc:648] HybridClock initialized: now 1750774694499771 us; error 57 us; skew 500 ppm
I20250624 14:18:14.500633 2118 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:18:14.513828 2118 webserver.cc:469] Webserver started at http://127.2.14.62:40817/ using document root <none> and password file <none>
I20250624 14:18:14.514858 2118 fs_manager.cc:362] Metadata directory not provided
I20250624 14:18:14.515065 2118 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:18:14.515491 2118 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:18:14.520159 2118 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "729b0c57174c4c428e04c11a2e756efd"
format_stamp: "Formatted at 2025-06-24 14:18:14 on dist-test-slave-hs17"
I20250624 14:18:14.521257 2118 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "729b0c57174c4c428e04c11a2e756efd"
format_stamp: "Formatted at 2025-06-24 14:18:14 on dist-test-slave-hs17"
I20250624 14:18:14.528725 2118 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250624 14:18:14.534515 2134 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:14.535756 2118 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250624 14:18:14.536118 2118 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
uuid: "729b0c57174c4c428e04c11a2e756efd"
format_stamp: "Formatted at 2025-06-24 14:18:14 on dist-test-slave-hs17"
I20250624 14:18:14.536456 2118 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:18:14.590324 2118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:18:14.591922 2118 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:18:14.592387 2118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:18:14.664456 2118 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.62:37963
I20250624 14:18:14.664565 2185 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.62:37963 every 8 connection(s)
I20250624 14:18:14.667237 2118 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 14:18:14.671663 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 2118
I20250624 14:18:14.672240 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 14:18:14.673792 2186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:14.694802 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd: Bootstrap starting.
I20250624 14:18:14.702575 2186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:14.705144 2186 log.cc:826] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd: Log is configured to *not* fsync() on all Append() calls
I20250624 14:18:14.710826 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd: No bootstrap required, opened a new log
I20250624 14:18:14.733170 2186 raft_consensus.cc:357] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b0c57174c4c428e04c11a2e756efd" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37963 } }
I20250624 14:18:14.734084 2186 raft_consensus.cc:383] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:18:14.734467 2186 raft_consensus.cc:738] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729b0c57174c4c428e04c11a2e756efd, State: Initialized, Role: FOLLOWER
I20250624 14:18:14.735625 2186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b0c57174c4c428e04c11a2e756efd" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37963 } }
I20250624 14:18:14.736327 2186 raft_consensus.cc:397] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:18:14.736630 2186 raft_consensus.cc:491] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:18:14.736950 2186 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:18:14.741318 2186 raft_consensus.cc:513] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b0c57174c4c428e04c11a2e756efd" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37963 } }
I20250624 14:18:14.742094 2186 leader_election.cc:304] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 729b0c57174c4c428e04c11a2e756efd; no voters:
I20250624 14:18:14.744153 2186 leader_election.cc:290] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 14:18:14.744835 2191 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:18:14.747123 2191 raft_consensus.cc:695] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 1 LEADER]: Becoming Leader. State: Replica: 729b0c57174c4c428e04c11a2e756efd, State: Running, Role: LEADER
I20250624 14:18:14.747932 2191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b0c57174c4c428e04c11a2e756efd" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37963 } }
I20250624 14:18:14.748950 2186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [sys.catalog]: configured and running, proceeding with master startup.
I20250624 14:18:14.759392 2192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "729b0c57174c4c428e04c11a2e756efd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b0c57174c4c428e04c11a2e756efd" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37963 } } }
I20250624 14:18:14.760107 2192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [sys.catalog]: This master's current role is: LEADER
I20250624 14:18:14.761348 2193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 729b0c57174c4c428e04c11a2e756efd. Latest consensus state: current_term: 1 leader_uuid: "729b0c57174c4c428e04c11a2e756efd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b0c57174c4c428e04c11a2e756efd" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37963 } } }
I20250624 14:18:14.761991 2193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [sys.catalog]: This master's current role is: LEADER
I20250624 14:18:14.766743 2200 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 14:18:14.780827 2200 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 14:18:14.797951 2200 catalog_manager.cc:1349] Generated new cluster ID: d756353d23264eac96bce96323bec805
I20250624 14:18:14.798283 2200 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 14:18:14.832139 2200 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 14:18:14.834249 2200 catalog_manager.cc:1506] Loading token signing keys...
I20250624 14:18:14.852629 2200 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd: Generated new TSK 0
I20250624 14:18:14.853803 2200 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 14:18:14.870335 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.1:0
--local_ip_for_outbound_sockets=127.2.14.1
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:37963
--builtin_ntp_servers=127.2.14.20:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 14:18:15.187398 2210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:15.187970 2210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:18:15.188472 2210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:15.220350 2210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:18:15.221220 2210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.1
I20250624 14:18:15.257596 2210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:37963
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.1
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:18:15.258947 2210 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:18:15.260643 2210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:18:15.278731 2217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:15.281571 2216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:16.465763 2218 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250624 14:18:16.467552 2219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:16.468200 2210 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:18:16.471262 2210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:18:16.474143 2210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:18:16.475618 2210 hybrid_clock.cc:648] HybridClock initialized: now 1750774696475551 us; error 80 us; skew 500 ppm
I20250624 14:18:16.476478 2210 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:18:16.484278 2210 webserver.cc:469] Webserver started at http://127.2.14.1:38683/ using document root <none> and password file <none>
I20250624 14:18:16.485348 2210 fs_manager.cc:362] Metadata directory not provided
I20250624 14:18:16.485574 2210 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:18:16.486064 2210 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:18:16.490839 2210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a8b705219c744c81b763be14f125c1ae"
format_stamp: "Formatted at 2025-06-24 14:18:16 on dist-test-slave-hs17"
I20250624 14:18:16.492120 2210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a8b705219c744c81b763be14f125c1ae"
format_stamp: "Formatted at 2025-06-24 14:18:16 on dist-test-slave-hs17"
I20250624 14:18:16.500427 2210 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20250624 14:18:16.506933 2226 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:16.508139 2210 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250624 14:18:16.508456 2210 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a8b705219c744c81b763be14f125c1ae"
format_stamp: "Formatted at 2025-06-24 14:18:16 on dist-test-slave-hs17"
I20250624 14:18:16.508785 2210 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:18:16.568478 2210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:18:16.570199 2210 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:18:16.570657 2210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:18:16.573287 2210 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:18:16.577752 2210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:18:16.577957 2210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:16.578245 2210 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:18:16.578400 2210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:16.717614 2210 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.1:33545
I20250624 14:18:16.717723 2338 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.1:33545 every 8 connection(s)
I20250624 14:18:16.720528 2210 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 14:18:16.731240 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 2210
I20250624 14:18:16.731686 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 14:18:16.738734 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.2:0
--local_ip_for_outbound_sockets=127.2.14.2
--webserver_interface=127.2.14.2
--webserver_port=0
--tserver_master_addrs=127.2.14.62:37963
--builtin_ntp_servers=127.2.14.20:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:18:16.743348 2339 heartbeater.cc:344] Connected to a master server at 127.2.14.62:37963
I20250624 14:18:16.743857 2339 heartbeater.cc:461] Registering TS with master...
I20250624 14:18:16.744951 2339 heartbeater.cc:507] Master 127.2.14.62:37963 requested a full tablet report, sending...
I20250624 14:18:16.747794 2151 ts_manager.cc:194] Registered new tserver with Master: a8b705219c744c81b763be14f125c1ae (127.2.14.1:33545)
I20250624 14:18:16.749812 2151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.1:47605
W20250624 14:18:17.042268 2343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:17.042752 2343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:18:17.043256 2343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:17.075146 2343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:18:17.075994 2343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.2
I20250624 14:18:17.110740 2343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.14.2
--webserver_port=0
--tserver_master_addrs=127.2.14.62:37963
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.2
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:18:17.112118 2343 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:18:17.113826 2343 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:18:17.130115 2349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:17.754086 2339 heartbeater.cc:499] Master 127.2.14.62:37963 was elected leader, sending a full tablet report...
W20250624 14:18:17.130152 2350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:17.132375 2352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:18.295164 2351 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 14:18:18.295277 2343 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:18:18.298972 2343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:18:18.301299 2343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:18:18.302695 2343 hybrid_clock.cc:648] HybridClock initialized: now 1750774698302649 us; error 52 us; skew 500 ppm
I20250624 14:18:18.303496 2343 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:18:18.309757 2343 webserver.cc:469] Webserver started at http://127.2.14.2:43001/ using document root <none> and password file <none>
I20250624 14:18:18.310760 2343 fs_manager.cc:362] Metadata directory not provided
I20250624 14:18:18.310988 2343 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:18:18.311439 2343 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:18:18.316017 2343 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "503b4a480aa94f2981c90eee7dae264d"
format_stamp: "Formatted at 2025-06-24 14:18:18 on dist-test-slave-hs17"
I20250624 14:18:18.317149 2343 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "503b4a480aa94f2981c90eee7dae264d"
format_stamp: "Formatted at 2025-06-24 14:18:18 on dist-test-slave-hs17"
I20250624 14:18:18.324398 2343 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250624 14:18:18.329963 2359 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:18.331053 2343 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250624 14:18:18.331360 2343 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "503b4a480aa94f2981c90eee7dae264d"
format_stamp: "Formatted at 2025-06-24 14:18:18 on dist-test-slave-hs17"
I20250624 14:18:18.331741 2343 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:18:18.391201 2343 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:18:18.392825 2343 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:18:18.393251 2343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:18:18.395771 2343 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:18:18.399832 2343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:18:18.400046 2343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:18.400285 2343 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:18:18.400444 2343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:18.531488 2343 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.2:44891
I20250624 14:18:18.531656 2471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.2:44891 every 8 connection(s)
I20250624 14:18:18.534188 2343 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 14:18:18.536006 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 2343
I20250624 14:18:18.536518 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 14:18:18.544919 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.3:0
--local_ip_for_outbound_sockets=127.2.14.3
--webserver_interface=127.2.14.3
--webserver_port=0
--tserver_master_addrs=127.2.14.62:37963
--builtin_ntp_servers=127.2.14.20:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:18:18.561460 2472 heartbeater.cc:344] Connected to a master server at 127.2.14.62:37963
I20250624 14:18:18.562036 2472 heartbeater.cc:461] Registering TS with master...
I20250624 14:18:18.563423 2472 heartbeater.cc:507] Master 127.2.14.62:37963 requested a full tablet report, sending...
I20250624 14:18:18.565992 2151 ts_manager.cc:194] Registered new tserver with Master: 503b4a480aa94f2981c90eee7dae264d (127.2.14.2:44891)
I20250624 14:18:18.567334 2151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.2:58533
W20250624 14:18:18.856462 2476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:18.856997 2476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:18:18.857488 2476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:18.889397 2476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:18:18.890265 2476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.3
I20250624 14:18:18.925313 2476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.14.3
--webserver_port=0
--tserver_master_addrs=127.2.14.62:37963
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.3
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:18:18.926648 2476 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:18:18.928366 2476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:18:18.944913 2483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:19.570691 2472 heartbeater.cc:499] Master 127.2.14.62:37963 was elected leader, sending a full tablet report...
W20250624 14:18:18.946045 2485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:18.945107 2482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:20.124559 2484 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 14:18:20.124668 2476 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:18:20.128521 2476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:18:20.130682 2476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:18:20.132042 2476 hybrid_clock.cc:648] HybridClock initialized: now 1750774700132001 us; error 61 us; skew 500 ppm
I20250624 14:18:20.132831 2476 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:18:20.139986 2476 webserver.cc:469] Webserver started at http://127.2.14.3:44793/ using document root <none> and password file <none>
I20250624 14:18:20.140935 2476 fs_manager.cc:362] Metadata directory not provided
I20250624 14:18:20.141152 2476 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:18:20.141607 2476 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:18:20.146325 2476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d5b48608f3924e5aafa414af2f04d672"
format_stamp: "Formatted at 2025-06-24 14:18:20 on dist-test-slave-hs17"
I20250624 14:18:20.147449 2476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d5b48608f3924e5aafa414af2f04d672"
format_stamp: "Formatted at 2025-06-24 14:18:20 on dist-test-slave-hs17"
I20250624 14:18:20.154918 2476 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250624 14:18:20.160769 2492 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:20.161906 2476 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250624 14:18:20.162240 2476 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d5b48608f3924e5aafa414af2f04d672"
format_stamp: "Formatted at 2025-06-24 14:18:20 on dist-test-slave-hs17"
I20250624 14:18:20.162647 2476 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:18:20.220616 2476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:18:20.222157 2476 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:18:20.222606 2476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:18:20.225198 2476 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:18:20.229341 2476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:18:20.229650 2476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:20.229885 2476 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:18:20.230021 2476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:20.368332 2476 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.3:41391
I20250624 14:18:20.368448 2604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.3:41391 every 8 connection(s)
I20250624 14:18:20.371095 2476 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 14:18:20.379434 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 2476
I20250624 14:18:20.380026 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 14:18:20.387428 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.4:0
--local_ip_for_outbound_sockets=127.2.14.4
--webserver_interface=127.2.14.4
--webserver_port=0
--tserver_master_addrs=127.2.14.62:37963
--builtin_ntp_servers=127.2.14.20:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:18:20.394814 2605 heartbeater.cc:344] Connected to a master server at 127.2.14.62:37963
I20250624 14:18:20.395385 2605 heartbeater.cc:461] Registering TS with master...
I20250624 14:18:20.396560 2605 heartbeater.cc:507] Master 127.2.14.62:37963 requested a full tablet report, sending...
I20250624 14:18:20.398947 2151 ts_manager.cc:194] Registered new tserver with Master: d5b48608f3924e5aafa414af2f04d672 (127.2.14.3:41391)
I20250624 14:18:20.400410 2151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.3:48479
W20250624 14:18:20.695370 2609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:20.695979 2609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:18:20.696513 2609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:20.728120 2609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:18:20.729007 2609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.4
I20250624 14:18:20.766736 2609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.14.4
--webserver_port=0
--tserver_master_addrs=127.2.14.62:37963
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.4
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:18:20.768224 2609 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:18:20.769963 2609 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:18:20.786382 2615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:21.403864 2605 heartbeater.cc:499] Master 127.2.14.62:37963 was elected leader, sending a full tablet report...
W20250624 14:18:20.786810 2618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:20.787206 2616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:20.789140 2609 server_base.cc:1048] running on GCE node
I20250624 14:18:21.970604 2609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:18:21.973407 2609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:18:21.974845 2609 hybrid_clock.cc:648] HybridClock initialized: now 1750774701974801 us; error 56 us; skew 500 ppm
I20250624 14:18:21.975693 2609 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:18:21.982412 2609 webserver.cc:469] Webserver started at http://127.2.14.4:40121/ using document root <none> and password file <none>
I20250624 14:18:21.983345 2609 fs_manager.cc:362] Metadata directory not provided
I20250624 14:18:21.983543 2609 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:18:21.984061 2609 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:18:21.988618 2609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b14b1e96e88f4996bad23d9377c00046"
format_stamp: "Formatted at 2025-06-24 14:18:21 on dist-test-slave-hs17"
I20250624 14:18:21.989722 2609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b14b1e96e88f4996bad23d9377c00046"
format_stamp: "Formatted at 2025-06-24 14:18:21 on dist-test-slave-hs17"
I20250624 14:18:21.997027 2609 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250624 14:18:22.002921 2626 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:22.004051 2609 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250624 14:18:22.004371 2609 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b14b1e96e88f4996bad23d9377c00046"
format_stamp: "Formatted at 2025-06-24 14:18:21 on dist-test-slave-hs17"
I20250624 14:18:22.004702 2609 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:18:22.068544 2609 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:18:22.070096 2609 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:18:22.070526 2609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:18:22.073056 2609 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:18:22.077163 2609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:18:22.077363 2609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:22.077634 2609 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:18:22.077787 2609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:22.219415 2609 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.4:43315
I20250624 14:18:22.219513 2738 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.4:43315 every 8 connection(s)
I20250624 14:18:22.222122 2609 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250624 14:18:22.225891 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 2609
I20250624 14:18:22.226533 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250624 14:18:22.244333 2739 heartbeater.cc:344] Connected to a master server at 127.2.14.62:37963
I20250624 14:18:22.244750 2739 heartbeater.cc:461] Registering TS with master...
I20250624 14:18:22.245889 2739 heartbeater.cc:507] Master 127.2.14.62:37963 requested a full tablet report, sending...
I20250624 14:18:22.248350 2151 ts_manager.cc:194] Registered new tserver with Master: b14b1e96e88f4996bad23d9377c00046 (127.2.14.4:43315)
I20250624 14:18:22.249565 2151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.4:49131
I20250624 14:18:22.249989 2104 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250624 14:18:22.291313 2151 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38946:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250624 14:18:22.365635 2540 tablet_service.cc:1468] Processing CreateTablet for tablet 1c42034c67bd4625b75333e47cf00f32 (DEFAULT_TABLE table=TestTable [id=17bc1ec8c9d04065b8a39d489c9bb294]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:18:22.366493 2674 tablet_service.cc:1468] Processing CreateTablet for tablet 1c42034c67bd4625b75333e47cf00f32 (DEFAULT_TABLE table=TestTable [id=17bc1ec8c9d04065b8a39d489c9bb294]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:18:22.367974 2540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c42034c67bd4625b75333e47cf00f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:22.367100 2407 tablet_service.cc:1468] Processing CreateTablet for tablet 1c42034c67bd4625b75333e47cf00f32 (DEFAULT_TABLE table=TestTable [id=17bc1ec8c9d04065b8a39d489c9bb294]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:18:22.368628 2674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c42034c67bd4625b75333e47cf00f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:22.369140 2407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c42034c67bd4625b75333e47cf00f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:22.405514 2760 tablet_bootstrap.cc:492] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672: Bootstrap starting.
I20250624 14:18:22.406271 2758 tablet_bootstrap.cc:492] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046: Bootstrap starting.
I20250624 14:18:22.414237 2758 tablet_bootstrap.cc:654] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:22.414950 2759 tablet_bootstrap.cc:492] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d: Bootstrap starting.
I20250624 14:18:22.417049 2758 log.cc:826] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046: Log is configured to *not* fsync() on all Append() calls
I20250624 14:18:22.422315 2760 tablet_bootstrap.cc:654] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:22.424798 2760 log.cc:826] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672: Log is configured to *not* fsync() on all Append() calls
I20250624 14:18:22.425307 2759 tablet_bootstrap.cc:654] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:22.427659 2758 tablet_bootstrap.cc:492] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046: No bootstrap required, opened a new log
I20250624 14:18:22.427702 2759 log.cc:826] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d: Log is configured to *not* fsync() on all Append() calls
I20250624 14:18:22.428223 2758 ts_tablet_manager.cc:1397] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.010s
I20250624 14:18:22.430167 2760 tablet_bootstrap.cc:492] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672: No bootstrap required, opened a new log
I20250624 14:18:22.430759 2760 ts_tablet_manager.cc:1397] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.004s
I20250624 14:18:22.433595 2759 tablet_bootstrap.cc:492] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d: No bootstrap required, opened a new log
I20250624 14:18:22.434165 2759 ts_tablet_manager.cc:1397] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.005s
I20250624 14:18:22.450186 2760 raft_consensus.cc:357] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
I20250624 14:18:22.451067 2760 raft_consensus.cc:383] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:18:22.451339 2760 raft_consensus.cc:738] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b48608f3924e5aafa414af2f04d672, State: Initialized, Role: FOLLOWER
I20250624 14:18:22.452193 2760 consensus_queue.cc:260] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
I20250624 14:18:22.456420 2760 ts_tablet_manager.cc:1428] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672: Time spent starting tablet: real 0.025s user 0.020s sys 0.005s
I20250624 14:18:22.456122 2758 raft_consensus.cc:357] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
I20250624 14:18:22.457135 2758 raft_consensus.cc:383] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:18:22.457489 2758 raft_consensus.cc:738] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14b1e96e88f4996bad23d9377c00046, State: Initialized, Role: FOLLOWER
I20250624 14:18:22.458577 2758 consensus_queue.cc:260] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
I20250624 14:18:22.462100 2739 heartbeater.cc:499] Master 127.2.14.62:37963 was elected leader, sending a full tablet report...
I20250624 14:18:22.463514 2758 ts_tablet_manager.cc:1428] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046: Time spent starting tablet: real 0.035s user 0.024s sys 0.009s
I20250624 14:18:22.463155 2759 raft_consensus.cc:357] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
I20250624 14:18:22.464506 2759 raft_consensus.cc:383] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:18:22.464829 2759 raft_consensus.cc:738] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 503b4a480aa94f2981c90eee7dae264d, State: Initialized, Role: FOLLOWER
I20250624 14:18:22.465890 2759 consensus_queue.cc:260] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
I20250624 14:18:22.472563 2759 ts_tablet_manager.cc:1428] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d: Time spent starting tablet: real 0.038s user 0.027s sys 0.011s
W20250624 14:18:22.477442 2740 tablet.cc:2378] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 14:18:22.543195 2473 tablet.cc:2378] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 14:18:22.627640 2606 tablet.cc:2378] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:18:22.660326 2765 raft_consensus.cc:491] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 14:18:22.660844 2765 raft_consensus.cc:513] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
I20250624 14:18:22.663343 2765 leader_election.cc:290] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5b48608f3924e5aafa414af2f04d672 (127.2.14.3:41391), 503b4a480aa94f2981c90eee7dae264d (127.2.14.2:44891)
I20250624 14:18:22.673452 2560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c42034c67bd4625b75333e47cf00f32" candidate_uuid: "b14b1e96e88f4996bad23d9377c00046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b48608f3924e5aafa414af2f04d672" is_pre_election: true
I20250624 14:18:22.674378 2560 raft_consensus.cc:2466] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b14b1e96e88f4996bad23d9377c00046 in term 0.
I20250624 14:18:22.675844 2627 leader_election.cc:304] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b14b1e96e88f4996bad23d9377c00046, d5b48608f3924e5aafa414af2f04d672; no voters:
I20250624 14:18:22.676731 2765 raft_consensus.cc:2802] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 14:18:22.677014 2765 raft_consensus.cc:491] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 14:18:22.676748 2427 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c42034c67bd4625b75333e47cf00f32" candidate_uuid: "b14b1e96e88f4996bad23d9377c00046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "503b4a480aa94f2981c90eee7dae264d" is_pre_election: true
I20250624 14:18:22.677270 2765 raft_consensus.cc:3058] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:18:22.677600 2427 raft_consensus.cc:2466] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b14b1e96e88f4996bad23d9377c00046 in term 0.
I20250624 14:18:22.681856 2765 raft_consensus.cc:513] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
I20250624 14:18:22.683379 2765 leader_election.cc:290] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [CANDIDATE]: Term 1 election: Requested vote from peers d5b48608f3924e5aafa414af2f04d672 (127.2.14.3:41391), 503b4a480aa94f2981c90eee7dae264d (127.2.14.2:44891)
I20250624 14:18:22.684252 2560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c42034c67bd4625b75333e47cf00f32" candidate_uuid: "b14b1e96e88f4996bad23d9377c00046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b48608f3924e5aafa414af2f04d672"
I20250624 14:18:22.684527 2427 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c42034c67bd4625b75333e47cf00f32" candidate_uuid: "b14b1e96e88f4996bad23d9377c00046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "503b4a480aa94f2981c90eee7dae264d"
I20250624 14:18:22.684813 2560 raft_consensus.cc:3058] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:18:22.685061 2427 raft_consensus.cc:3058] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:18:22.692063 2427 raft_consensus.cc:2466] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b14b1e96e88f4996bad23d9377c00046 in term 1.
I20250624 14:18:22.692054 2560 raft_consensus.cc:2466] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b14b1e96e88f4996bad23d9377c00046 in term 1.
I20250624 14:18:22.693256 2627 leader_election.cc:304] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b14b1e96e88f4996bad23d9377c00046, d5b48608f3924e5aafa414af2f04d672; no voters:
I20250624 14:18:22.693933 2765 raft_consensus.cc:2802] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:18:22.695696 2765 raft_consensus.cc:695] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [term 1 LEADER]: Becoming Leader. State: Replica: b14b1e96e88f4996bad23d9377c00046, State: Running, Role: LEADER
I20250624 14:18:22.696928 2765 consensus_queue.cc:237] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
I20250624 14:18:22.710673 2151 catalog_manager.cc:5582] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 reported cstate change: term changed from 0 to 1, leader changed from <none> to b14b1e96e88f4996bad23d9377c00046 (127.2.14.4). New cstate: current_term: 1 leader_uuid: "b14b1e96e88f4996bad23d9377c00046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } health_report { overall_health: HEALTHY } } }
I20250624 14:18:22.743850 2104 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250624 14:18:22.747973 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 503b4a480aa94f2981c90eee7dae264d to finish bootstrapping
I20250624 14:18:22.763376 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d5b48608f3924e5aafa414af2f04d672 to finish bootstrapping
I20250624 14:18:22.776510 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b14b1e96e88f4996bad23d9377c00046 to finish bootstrapping
I20250624 14:18:22.789032 2104 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250624 14:18:22.792498 2104 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "1c42034c67bd4625b75333e47cf00f32"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: LEADER
}
}
ts_infos {
permanent_uuid: "d5b48608f3924e5aafa414af2f04d672"
rpc_addresses {
host: "127.2.14.3"
port: 41391
}
}
ts_infos {
permanent_uuid: "503b4a480aa94f2981c90eee7dae264d"
rpc_addresses {
host: "127.2.14.2"
port: 44891
}
}
ts_infos {
permanent_uuid: "b14b1e96e88f4996bad23d9377c00046"
rpc_addresses {
host: "127.2.14.4"
port: 43315
}
}
I20250624 14:18:23.217754 2773 consensus_queue.cc:1035] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 14:18:23.241356 2776 consensus_queue.cc:1035] T 1c42034c67bd4625b75333e47cf00f32 P b14b1e96e88f4996bad23d9377c00046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 14:18:23.244109 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 2609
W20250624 14:18:23.272961 2363 connection.cc:537] server connection from 127.2.14.4:53213 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 14:18:23.273613 2138 connection.cc:537] server connection from 127.2.14.4:49131 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250624 14:18:23.274556 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 2118
I20250624 14:18:23.302623 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:37963
--webserver_interface=127.2.14.62
--webserver_port=40817
--builtin_ntp_servers=127.2.14.20:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.14.62:37963 with env {}
W20250624 14:18:23.647064 2783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:23.647672 2783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:18:23.648089 2783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:23.680055 2783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 14:18:23.680367 2783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:18:23.680575 2783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 14:18:23.680785 2783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 14:18:23.717212 2783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.14.62:37963
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:37963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.14.62
--webserver_port=40817
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:18:23.718621 2783 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:18:23.720402 2783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:18:23.737398 2792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:23.780599 2339 heartbeater.cc:646] Failed to heartbeat to 127.2.14.62:37963 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.14.62:37963: connect: Connection refused (error 111)
W20250624 14:18:24.244813 2605 heartbeater.cc:646] Failed to heartbeat to 127.2.14.62:37963 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.14.62:37963: connect: Connection refused (error 111)
W20250624 14:18:24.269028 2472 heartbeater.cc:646] Failed to heartbeat to 127.2.14.62:37963 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.14.62:37963: connect: Connection refused (error 111)
I20250624 14:18:24.734885 2798 raft_consensus.cc:491] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b14b1e96e88f4996bad23d9377c00046)
I20250624 14:18:24.735306 2798 raft_consensus.cc:513] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
I20250624 14:18:24.737480 2798 leader_election.cc:290] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 503b4a480aa94f2981c90eee7dae264d (127.2.14.2:44891), b14b1e96e88f4996bad23d9377c00046 (127.2.14.4:43315)
W20250624 14:18:24.741261 2493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.14.4:43315: connect: Connection refused (error 111)
W20250624 14:18:24.745498 2493 leader_election.cc:336] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b14b1e96e88f4996bad23d9377c00046 (127.2.14.4:43315): Network error: Client connection negotiation failed: client connection to 127.2.14.4:43315: connect: Connection refused (error 111)
I20250624 14:18:24.748872 2803 raft_consensus.cc:491] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b14b1e96e88f4996bad23d9377c00046)
I20250624 14:18:24.749347 2803 raft_consensus.cc:513] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
I20250624 14:18:24.751533 2427 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c42034c67bd4625b75333e47cf00f32" candidate_uuid: "d5b48608f3924e5aafa414af2f04d672" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "503b4a480aa94f2981c90eee7dae264d" is_pre_election: true
I20250624 14:18:24.751873 2803 leader_election.cc:290] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5b48608f3924e5aafa414af2f04d672 (127.2.14.3:41391), b14b1e96e88f4996bad23d9377c00046 (127.2.14.4:43315)
I20250624 14:18:24.752141 2427 raft_consensus.cc:2466] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b48608f3924e5aafa414af2f04d672 in term 1.
W20250624 14:18:24.756028 2360 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.14.4:43315: connect: Connection refused (error 111)
I20250624 14:18:24.755931 2496 leader_election.cc:304] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 503b4a480aa94f2981c90eee7dae264d, d5b48608f3924e5aafa414af2f04d672; no voters: b14b1e96e88f4996bad23d9377c00046
I20250624 14:18:24.757242 2798 raft_consensus.cc:2802] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250624 14:18:24.757602 2798 raft_consensus.cc:491] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b14b1e96e88f4996bad23d9377c00046)
I20250624 14:18:24.757936 2798 raft_consensus.cc:3058] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 1 FOLLOWER]: Advancing to term 2
I20250624 14:18:24.766088 2798 raft_consensus.cc:513] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
I20250624 14:18:24.769382 2427 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c42034c67bd4625b75333e47cf00f32" candidate_uuid: "d5b48608f3924e5aafa414af2f04d672" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "503b4a480aa94f2981c90eee7dae264d"
I20250624 14:18:24.769953 2427 raft_consensus.cc:3058] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 1 FOLLOWER]: Advancing to term 2
I20250624 14:18:24.771644 2560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c42034c67bd4625b75333e47cf00f32" candidate_uuid: "503b4a480aa94f2981c90eee7dae264d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d5b48608f3924e5aafa414af2f04d672" is_pre_election: true
I20250624 14:18:24.772197 2560 raft_consensus.cc:2391] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 503b4a480aa94f2981c90eee7dae264d in current term 2: Already voted for candidate d5b48608f3924e5aafa414af2f04d672 in this term.
I20250624 14:18:24.774614 2798 leader_election.cc:290] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [CANDIDATE]: Term 2 election: Requested vote from peers 503b4a480aa94f2981c90eee7dae264d (127.2.14.2:44891), b14b1e96e88f4996bad23d9377c00046 (127.2.14.4:43315)
W20250624 14:18:24.775172 2360 leader_election.cc:336] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b14b1e96e88f4996bad23d9377c00046 (127.2.14.4:43315): Network error: Client connection negotiation failed: client connection to 127.2.14.4:43315: connect: Connection refused (error 111)
I20250624 14:18:24.775755 2360 leader_election.cc:304] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 503b4a480aa94f2981c90eee7dae264d; no voters: b14b1e96e88f4996bad23d9377c00046, d5b48608f3924e5aafa414af2f04d672
I20250624 14:18:24.776381 2427 raft_consensus.cc:2466] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b48608f3924e5aafa414af2f04d672 in term 2.
I20250624 14:18:24.777441 2803 raft_consensus.cc:2747] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250624 14:18:24.777976 2496 leader_election.cc:304] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 503b4a480aa94f2981c90eee7dae264d, d5b48608f3924e5aafa414af2f04d672; no voters:
I20250624 14:18:24.778785 2798 raft_consensus.cc:2802] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 2 FOLLOWER]: Leader election won for term 2
W20250624 14:18:24.778889 2493 leader_election.cc:336] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b14b1e96e88f4996bad23d9377c00046 (127.2.14.4:43315): Network error: Client connection negotiation failed: client connection to 127.2.14.4:43315: connect: Connection refused (error 111)
I20250624 14:18:24.780944 2798 raft_consensus.cc:695] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 2 LEADER]: Becoming Leader. State: Replica: d5b48608f3924e5aafa414af2f04d672, State: Running, Role: LEADER
I20250624 14:18:24.781945 2798 consensus_queue.cc:237] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
W20250624 14:18:23.737469 2790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:23.739645 2783 server_base.cc:1048] running on GCE node
W20250624 14:18:23.737586 2789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:24.958669 2783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:18:24.961442 2783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:18:24.962814 2783 hybrid_clock.cc:648] HybridClock initialized: now 1750774704962771 us; error 58 us; skew 500 ppm
I20250624 14:18:24.963701 2783 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:18:24.972924 2783 webserver.cc:469] Webserver started at http://127.2.14.62:40817/ using document root <none> and password file <none>
I20250624 14:18:24.974684 2783 fs_manager.cc:362] Metadata directory not provided
I20250624 14:18:24.975095 2783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:18:24.987493 2783 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.002s sys 0.005s
I20250624 14:18:24.992712 2814 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:24.993952 2783 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250624 14:18:24.994308 2783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
uuid: "729b0c57174c4c428e04c11a2e756efd"
format_stamp: "Formatted at 2025-06-24 14:18:14 on dist-test-slave-hs17"
I20250624 14:18:24.996513 2783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:18:25.068310 2783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:18:25.070048 2783 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:18:25.070680 2783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:18:25.158612 2783 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.62:37963
I20250624 14:18:25.158758 2869 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.62:37963 every 8 connection(s)
I20250624 14:18:25.162254 2783 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 14:18:25.163257 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 2783
I20250624 14:18:25.164000 2104 kudu-admin-test.cc:735] Forcing unsafe config change on tserver d5b48608f3924e5aafa414af2f04d672
I20250624 14:18:25.172269 2427 raft_consensus.cc:1273] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 2 FOLLOWER]: Refusing update from remote peer d5b48608f3924e5aafa414af2f04d672: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250624 14:18:25.174443 2798 consensus_queue.cc:1035] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250624 14:18:25.178352 2870 sys_catalog.cc:263] Verifying existing consensus state
I20250624 14:18:25.187143 2870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd: Bootstrap starting.
I20250624 14:18:25.211656 2472 heartbeater.cc:344] Connected to a master server at 127.2.14.62:37963
W20250624 14:18:25.234220 2493 consensus_peers.cc:487] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 -> Peer b14b1e96e88f4996bad23d9377c00046 (127.2.14.4:43315): Couldn't send request to peer b14b1e96e88f4996bad23d9377c00046. Status: Network error: Client connection negotiation failed: client connection to 127.2.14.4:43315: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250624 14:18:25.246376 2605 heartbeater.cc:344] Connected to a master server at 127.2.14.62:37963
I20250624 14:18:25.254971 2870 log.cc:826] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd: Log is configured to *not* fsync() on all Append() calls
I20250624 14:18:25.286549 2870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:18:25.287688 2870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd: Bootstrap complete.
I20250624 14:18:25.317770 2870 raft_consensus.cc:357] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b0c57174c4c428e04c11a2e756efd" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37963 } }
I20250624 14:18:25.321549 2870 raft_consensus.cc:738] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729b0c57174c4c428e04c11a2e756efd, State: Initialized, Role: FOLLOWER
I20250624 14:18:25.322629 2870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b0c57174c4c428e04c11a2e756efd" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37963 } }
I20250624 14:18:25.323452 2870 raft_consensus.cc:397] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:18:25.323870 2870 raft_consensus.cc:491] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:18:25.324314 2870 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 1 FOLLOWER]: Advancing to term 2
I20250624 14:18:25.336565 2870 raft_consensus.cc:513] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b0c57174c4c428e04c11a2e756efd" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37963 } }
I20250624 14:18:25.337682 2870 leader_election.cc:304] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 729b0c57174c4c428e04c11a2e756efd; no voters:
I20250624 14:18:25.340143 2870 leader_election.cc:290] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 14:18:25.340788 2880 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 2 FOLLOWER]: Leader election won for term 2
I20250624 14:18:25.347702 2870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [sys.catalog]: configured and running, proceeding with master startup.
I20250624 14:18:25.352218 2880 raft_consensus.cc:695] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [term 2 LEADER]: Becoming Leader. State: Replica: 729b0c57174c4c428e04c11a2e756efd, State: Running, Role: LEADER
I20250624 14:18:25.353446 2880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b0c57174c4c428e04c11a2e756efd" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37963 } }
I20250624 14:18:25.363832 2882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 729b0c57174c4c428e04c11a2e756efd. Latest consensus state: current_term: 2 leader_uuid: "729b0c57174c4c428e04c11a2e756efd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b0c57174c4c428e04c11a2e756efd" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37963 } } }
I20250624 14:18:25.364758 2882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [sys.catalog]: This master's current role is: LEADER
I20250624 14:18:25.363837 2881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "729b0c57174c4c428e04c11a2e756efd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b0c57174c4c428e04c11a2e756efd" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37963 } } }
I20250624 14:18:25.365494 2881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd [sys.catalog]: This master's current role is: LEADER
I20250624 14:18:25.377018 2887 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 14:18:25.390583 2887 catalog_manager.cc:671] Loaded metadata for table TestTable [id=17bc1ec8c9d04065b8a39d489c9bb294]
I20250624 14:18:25.399223 2887 tablet_loader.cc:96] loaded metadata for tablet 1c42034c67bd4625b75333e47cf00f32 (table TestTable [id=17bc1ec8c9d04065b8a39d489c9bb294])
I20250624 14:18:25.401202 2887 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 14:18:25.406716 2887 catalog_manager.cc:1261] Loaded cluster ID: d756353d23264eac96bce96323bec805
I20250624 14:18:25.407131 2887 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 14:18:25.415853 2887 catalog_manager.cc:1506] Loading token signing keys...
I20250624 14:18:25.421264 2887 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 729b0c57174c4c428e04c11a2e756efd: Loaded TSK: 0
I20250624 14:18:25.423013 2887 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250624 14:18:25.598865 2873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:25.599493 2873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:25.633181 2873 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250624 14:18:25.826619 2339 heartbeater.cc:344] Connected to a master server at 127.2.14.62:37963
I20250624 14:18:25.835635 2833 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "a8b705219c744c81b763be14f125c1ae" instance_seqno: 1750774696685173) as {username='slave'} at 127.2.14.1:36257; Asking this server to re-register.
I20250624 14:18:25.837944 2339 heartbeater.cc:461] Registering TS with master...
I20250624 14:18:25.838888 2339 heartbeater.cc:507] Master 127.2.14.62:37963 requested a full tablet report, sending...
I20250624 14:18:25.843240 2831 ts_manager.cc:194] Registered new tserver with Master: a8b705219c744c81b763be14f125c1ae (127.2.14.1:33545)
I20250624 14:18:26.242074 2833 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" instance_seqno: 1750774698500040) as {username='slave'} at 127.2.14.2:56157; Asking this server to re-register.
I20250624 14:18:26.244266 2472 heartbeater.cc:461] Registering TS with master...
I20250624 14:18:26.244958 2472 heartbeater.cc:507] Master 127.2.14.62:37963 requested a full tablet report, sending...
I20250624 14:18:26.248944 2833 ts_manager.cc:194] Registered new tserver with Master: 503b4a480aa94f2981c90eee7dae264d (127.2.14.2:44891)
I20250624 14:18:26.254400 2831 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" instance_seqno: 1750774700335475) as {username='slave'} at 127.2.14.3:33469; Asking this server to re-register.
I20250624 14:18:26.256424 2605 heartbeater.cc:461] Registering TS with master...
I20250624 14:18:26.257138 2605 heartbeater.cc:507] Master 127.2.14.62:37963 requested a full tablet report, sending...
I20250624 14:18:26.261073 2831 ts_manager.cc:194] Registered new tserver with Master: d5b48608f3924e5aafa414af2f04d672 (127.2.14.3:41391)
I20250624 14:18:26.258419 2833 catalog_manager.cc:5582] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d reported cstate change: term changed from 1 to 2, leader changed from b14b1e96e88f4996bad23d9377c00046 (127.2.14.4) to d5b48608f3924e5aafa414af2f04d672 (127.2.14.3). New cstate: current_term: 2 leader_uuid: "d5b48608f3924e5aafa414af2f04d672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } } }
W20250624 14:18:27.081722 2873 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.402s user 0.470s sys 0.894s
W20250624 14:18:27.082147 2873 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.402s user 0.470s sys 0.894s
I20250624 14:18:27.159098 2560 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "d5b48608f3924e5aafa414af2f04d672"
tablet_id: "1c42034c67bd4625b75333e47cf00f32"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "503b4a480aa94f2981c90eee7dae264d"
}
peers {
permanent_uuid: "d5b48608f3924e5aafa414af2f04d672"
}
}
from {username='slave'} at 127.0.0.1:39606
W20250624 14:18:27.160674 2560 raft_consensus.cc:2216] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } unsafe_config_change: true
I20250624 14:18:27.161957 2560 raft_consensus.cc:3053] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 2 LEADER]: Stepping down as leader of term 2
I20250624 14:18:27.162262 2560 raft_consensus.cc:738] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5b48608f3924e5aafa414af2f04d672, State: Running, Role: LEADER
I20250624 14:18:27.163408 2560 consensus_queue.cc:260] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
I20250624 14:18:27.164902 2560 raft_consensus.cc:3058] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 2 FOLLOWER]: Advancing to term 3
I20250624 14:18:28.376552 2928 raft_consensus.cc:491] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d5b48608f3924e5aafa414af2f04d672)
I20250624 14:18:28.377113 2928 raft_consensus.cc:513] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } }
I20250624 14:18:28.378763 2928 leader_election.cc:290] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d5b48608f3924e5aafa414af2f04d672 (127.2.14.3:41391), b14b1e96e88f4996bad23d9377c00046 (127.2.14.4:43315)
I20250624 14:18:28.379513 2560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c42034c67bd4625b75333e47cf00f32" candidate_uuid: "503b4a480aa94f2981c90eee7dae264d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d5b48608f3924e5aafa414af2f04d672" is_pre_election: true
W20250624 14:18:28.384128 2360 leader_election.cc:336] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b14b1e96e88f4996bad23d9377c00046 (127.2.14.4:43315): Network error: Client connection negotiation failed: client connection to 127.2.14.4:43315: connect: Connection refused (error 111)
I20250624 14:18:28.384512 2360 leader_election.cc:304] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 503b4a480aa94f2981c90eee7dae264d; no voters: b14b1e96e88f4996bad23d9377c00046, d5b48608f3924e5aafa414af2f04d672
I20250624 14:18:28.385218 2928 raft_consensus.cc:2747] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250624 14:18:28.675292 2931 raft_consensus.cc:491] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250624 14:18:28.675743 2931 raft_consensus.cc:513] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } unsafe_config_change: true
I20250624 14:18:28.676852 2931 leader_election.cc:290] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 503b4a480aa94f2981c90eee7dae264d (127.2.14.2:44891)
I20250624 14:18:28.678222 2427 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c42034c67bd4625b75333e47cf00f32" candidate_uuid: "d5b48608f3924e5aafa414af2f04d672" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "503b4a480aa94f2981c90eee7dae264d" is_pre_election: true
I20250624 14:18:28.678800 2427 raft_consensus.cc:2466] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b48608f3924e5aafa414af2f04d672 in term 2.
I20250624 14:18:28.679886 2496 leader_election.cc:304] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 503b4a480aa94f2981c90eee7dae264d, d5b48608f3924e5aafa414af2f04d672; no voters:
I20250624 14:18:28.680496 2931 raft_consensus.cc:2802] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250624 14:18:28.680733 2931 raft_consensus.cc:491] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250624 14:18:28.681084 2931 raft_consensus.cc:3058] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 3 FOLLOWER]: Advancing to term 4
I20250624 14:18:28.685519 2931 raft_consensus.cc:513] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } unsafe_config_change: true
I20250624 14:18:28.686631 2931 leader_election.cc:290] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [CANDIDATE]: Term 4 election: Requested vote from peers 503b4a480aa94f2981c90eee7dae264d (127.2.14.2:44891)
I20250624 14:18:28.687872 2427 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c42034c67bd4625b75333e47cf00f32" candidate_uuid: "d5b48608f3924e5aafa414af2f04d672" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "503b4a480aa94f2981c90eee7dae264d"
I20250624 14:18:28.688303 2427 raft_consensus.cc:3058] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 2 FOLLOWER]: Advancing to term 4
I20250624 14:18:28.693006 2427 raft_consensus.cc:2466] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b48608f3924e5aafa414af2f04d672 in term 4.
I20250624 14:18:28.694017 2496 leader_election.cc:304] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 503b4a480aa94f2981c90eee7dae264d, d5b48608f3924e5aafa414af2f04d672; no voters:
I20250624 14:18:28.694665 2931 raft_consensus.cc:2802] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 4 FOLLOWER]: Leader election won for term 4
I20250624 14:18:28.695535 2931 raft_consensus.cc:695] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 4 LEADER]: Becoming Leader. State: Replica: d5b48608f3924e5aafa414af2f04d672, State: Running, Role: LEADER
I20250624 14:18:28.696169 2931 consensus_queue.cc:237] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } unsafe_config_change: true
I20250624 14:18:28.701640 2833 catalog_manager.cc:5582] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER d5b48608f3924e5aafa414af2f04d672 (127.2.14.3), VOTER 503b4a480aa94f2981c90eee7dae264d (127.2.14.2). New cstate: current_term: 4 leader_uuid: "d5b48608f3924e5aafa414af2f04d672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b14b1e96e88f4996bad23d9377c00046" member_type: VOTER last_known_addr { host: "127.2.14.4" port: 43315 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } unsafe_config_change: true }
I20250624 14:18:29.116171 2427 raft_consensus.cc:1273] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 4 FOLLOWER]: Refusing update from remote peer d5b48608f3924e5aafa414af2f04d672: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250624 14:18:29.117501 2931 consensus_queue.cc:1035] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250624 14:18:29.125633 2932 raft_consensus.cc:2953] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER b14b1e96e88f4996bad23d9377c00046 (127.2.14.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } unsafe_config_change: true }
I20250624 14:18:29.127930 2427 raft_consensus.cc:2953] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER b14b1e96e88f4996bad23d9377c00046 (127.2.14.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } unsafe_config_change: true }
I20250624 14:18:29.138000 2833 catalog_manager.cc:5582] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 reported cstate change: config changed from index -1 to 3, VOTER b14b1e96e88f4996bad23d9377c00046 (127.2.14.4) evicted, no longer has a pending config: VOTER d5b48608f3924e5aafa414af2f04d672 (127.2.14.3), VOTER 503b4a480aa94f2981c90eee7dae264d (127.2.14.2). New cstate: current_term: 4 leader_uuid: "d5b48608f3924e5aafa414af2f04d672" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250624 14:18:29.145344 2833 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 1c42034c67bd4625b75333e47cf00f32 on TS b14b1e96e88f4996bad23d9377c00046: Not found: failed to reset TS proxy: Could not find TS for UUID b14b1e96e88f4996bad23d9377c00046
I20250624 14:18:29.162510 2560 consensus_queue.cc:237] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "a8b705219c744c81b763be14f125c1ae" member_type: NON_VOTER last_known_addr { host: "127.2.14.1" port: 33545 } attrs { promote: true } } unsafe_config_change: true
I20250624 14:18:29.168262 2427 raft_consensus.cc:1273] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 4 FOLLOWER]: Refusing update from remote peer d5b48608f3924e5aafa414af2f04d672: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250624 14:18:29.170145 2932 consensus_queue.cc:1035] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250624 14:18:29.180433 2932 raft_consensus.cc:2953] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER a8b705219c744c81b763be14f125c1ae (127.2.14.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "a8b705219c744c81b763be14f125c1ae" member_type: NON_VOTER last_known_addr { host: "127.2.14.1" port: 33545 } attrs { promote: true } } unsafe_config_change: true }
I20250624 14:18:29.184114 2426 raft_consensus.cc:2953] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER a8b705219c744c81b763be14f125c1ae (127.2.14.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "a8b705219c744c81b763be14f125c1ae" member_type: NON_VOTER last_known_addr { host: "127.2.14.1" port: 33545 } attrs { promote: true } } unsafe_config_change: true }
I20250624 14:18:29.193578 2815 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1c42034c67bd4625b75333e47cf00f32 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250624 14:18:29.197770 2817 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 1c42034c67bd4625b75333e47cf00f32 on TS b14b1e96e88f4996bad23d9377c00046 failed: Not found: failed to reset TS proxy: Could not find TS for UUID b14b1e96e88f4996bad23d9377c00046
I20250624 14:18:29.196992 2831 catalog_manager.cc:5582] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 reported cstate change: config changed from index 3 to 5, NON_VOTER a8b705219c744c81b763be14f125c1ae (127.2.14.1) added. New cstate: current_term: 4 leader_uuid: "d5b48608f3924e5aafa414af2f04d672" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8b705219c744c81b763be14f125c1ae" member_type: NON_VOTER last_known_addr { host: "127.2.14.1" port: 33545 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250624 14:18:29.201153 2494 consensus_peers.cc:487] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 -> Peer a8b705219c744c81b763be14f125c1ae (127.2.14.1:33545): Couldn't send request to peer a8b705219c744c81b763be14f125c1ae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c42034c67bd4625b75333e47cf00f32. This is attempt 1: this message will repeat every 5th retry.
I20250624 14:18:29.552718 2948 ts_tablet_manager.cc:927] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae: Initiating tablet copy from peer d5b48608f3924e5aafa414af2f04d672 (127.2.14.3:41391)
I20250624 14:18:29.555651 2948 tablet_copy_client.cc:323] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae: tablet copy: Beginning tablet copy session from remote peer at address 127.2.14.3:41391
I20250624 14:18:29.568260 2580 tablet_copy_service.cc:140] P d5b48608f3924e5aafa414af2f04d672: Received BeginTabletCopySession request for tablet 1c42034c67bd4625b75333e47cf00f32 from peer a8b705219c744c81b763be14f125c1ae ({username='slave'} at 127.2.14.1:56709)
I20250624 14:18:29.568719 2580 tablet_copy_service.cc:161] P d5b48608f3924e5aafa414af2f04d672: Beginning new tablet copy session on tablet 1c42034c67bd4625b75333e47cf00f32 from peer a8b705219c744c81b763be14f125c1ae at {username='slave'} at 127.2.14.1:56709: session id = a8b705219c744c81b763be14f125c1ae-1c42034c67bd4625b75333e47cf00f32
I20250624 14:18:29.574015 2580 tablet_copy_source_session.cc:215] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 14:18:29.579489 2948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c42034c67bd4625b75333e47cf00f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:29.600481 2948 tablet_copy_client.cc:806] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae: tablet copy: Starting download of 0 data blocks...
I20250624 14:18:29.601074 2948 tablet_copy_client.cc:670] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae: tablet copy: Starting download of 1 WAL segments...
I20250624 14:18:29.605116 2948 tablet_copy_client.cc:538] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 14:18:29.611338 2948 tablet_bootstrap.cc:492] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae: Bootstrap starting.
I20250624 14:18:29.624020 2948 log.cc:826] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae: Log is configured to *not* fsync() on all Append() calls
I20250624 14:18:29.634917 2948 tablet_bootstrap.cc:492] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:18:29.635869 2948 tablet_bootstrap.cc:492] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae: Bootstrap complete.
I20250624 14:18:29.636466 2948 ts_tablet_manager.cc:1397] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.004s
I20250624 14:18:29.653875 2948 raft_consensus.cc:357] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "a8b705219c744c81b763be14f125c1ae" member_type: NON_VOTER last_known_addr { host: "127.2.14.1" port: 33545 } attrs { promote: true } } unsafe_config_change: true
I20250624 14:18:29.654791 2948 raft_consensus.cc:738] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: a8b705219c744c81b763be14f125c1ae, State: Initialized, Role: LEARNER
I20250624 14:18:29.655397 2948 consensus_queue.cc:260] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "a8b705219c744c81b763be14f125c1ae" member_type: NON_VOTER last_known_addr { host: "127.2.14.1" port: 33545 } attrs { promote: true } } unsafe_config_change: true
I20250624 14:18:29.658600 2948 ts_tablet_manager.cc:1428] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae: Time spent starting tablet: real 0.022s user 0.017s sys 0.004s
I20250624 14:18:29.660393 2580 tablet_copy_service.cc:342] P d5b48608f3924e5aafa414af2f04d672: Request end of tablet copy session a8b705219c744c81b763be14f125c1ae-1c42034c67bd4625b75333e47cf00f32 received from {username='slave'} at 127.2.14.1:56709
I20250624 14:18:29.660779 2580 tablet_copy_service.cc:434] P d5b48608f3924e5aafa414af2f04d672: ending tablet copy session a8b705219c744c81b763be14f125c1ae-1c42034c67bd4625b75333e47cf00f32 on tablet 1c42034c67bd4625b75333e47cf00f32 with peer a8b705219c744c81b763be14f125c1ae
I20250624 14:18:30.141417 2294 raft_consensus.cc:1215] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250624 14:18:30.364602 2817 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 1c42034c67bd4625b75333e47cf00f32 on TS b14b1e96e88f4996bad23d9377c00046 failed: Not found: failed to reset TS proxy: Could not find TS for UUID b14b1e96e88f4996bad23d9377c00046
I20250624 14:18:30.591512 2956 raft_consensus.cc:1062] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672: attempting to promote NON_VOTER a8b705219c744c81b763be14f125c1ae to VOTER
I20250624 14:18:30.593138 2956 consensus_queue.cc:237] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "a8b705219c744c81b763be14f125c1ae" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 33545 } attrs { promote: false } } unsafe_config_change: true
I20250624 14:18:30.598368 2426 raft_consensus.cc:1273] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 4 FOLLOWER]: Refusing update from remote peer d5b48608f3924e5aafa414af2f04d672: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250624 14:18:30.598455 2294 raft_consensus.cc:1273] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae [term 4 LEARNER]: Refusing update from remote peer d5b48608f3924e5aafa414af2f04d672: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250624 14:18:30.599748 2954 consensus_queue.cc:1035] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250624 14:18:30.600595 2953 consensus_queue.cc:1035] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8b705219c744c81b763be14f125c1ae" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 33545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250624 14:18:30.607519 2954 raft_consensus.cc:2953] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, a8b705219c744c81b763be14f125c1ae (127.2.14.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "a8b705219c744c81b763be14f125c1ae" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 33545 } attrs { promote: false } } unsafe_config_change: true }
I20250624 14:18:30.609475 2426 raft_consensus.cc:2953] T 1c42034c67bd4625b75333e47cf00f32 P 503b4a480aa94f2981c90eee7dae264d [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, a8b705219c744c81b763be14f125c1ae (127.2.14.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "a8b705219c744c81b763be14f125c1ae" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 33545 } attrs { promote: false } } unsafe_config_change: true }
I20250624 14:18:30.614089 2294 raft_consensus.cc:2953] T 1c42034c67bd4625b75333e47cf00f32 P a8b705219c744c81b763be14f125c1ae [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, a8b705219c744c81b763be14f125c1ae (127.2.14.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } } peers { permanent_uuid: "a8b705219c744c81b763be14f125c1ae" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 33545 } attrs { promote: false } } unsafe_config_change: true }
I20250624 14:18:30.618683 2831 catalog_manager.cc:5582] T 1c42034c67bd4625b75333e47cf00f32 P d5b48608f3924e5aafa414af2f04d672 reported cstate change: config changed from index 5 to 6, a8b705219c744c81b763be14f125c1ae (127.2.14.1) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "d5b48608f3924e5aafa414af2f04d672" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d5b48608f3924e5aafa414af2f04d672" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "503b4a480aa94f2981c90eee7dae264d" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8b705219c744c81b763be14f125c1ae" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 33545 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250624 14:18:30.745417 2104 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250624 14:18:30.765074 2104 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "1c42034c67bd4625b75333e47cf00f32"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "d5b48608f3924e5aafa414af2f04d672"
rpc_addresses {
host: "127.2.14.3"
port: 41391
}
}
ts_infos {
permanent_uuid: "503b4a480aa94f2981c90eee7dae264d"
rpc_addresses {
host: "127.2.14.2"
port: 44891
}
}
ts_infos {
permanent_uuid: "a8b705219c744c81b763be14f125c1ae"
rpc_addresses {
host: "127.2.14.1"
port: 33545
}
}
I20250624 14:18:30.767331 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 2210
I20250624 14:18:30.793272 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 2343
I20250624 14:18:30.819352 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 2476
I20250624 14:18:30.849001 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 2783
2025-06-24T14:18:30Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19269 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250624 14:18:30.912007 2104 test_util.cc:276] Using random seed: 12174315
I20250624 14:18:30.918040 2104 ts_itest-base.cc:115] Starting cluster with:
I20250624 14:18:30.918241 2104 ts_itest-base.cc:116] --------------
I20250624 14:18:30.918391 2104 ts_itest-base.cc:117] 3 tablet servers
I20250624 14:18:30.918531 2104 ts_itest-base.cc:118] 3 replicas per TS
I20250624 14:18:30.918664 2104 ts_itest-base.cc:119] --------------
2025-06-24T14:18:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T14:18:30Z Disabled control of system clock
I20250624 14:18:30.959268 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:46289
--webserver_interface=127.2.14.62
--webserver_port=0
--builtin_ntp_servers=127.2.14.20:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.14.62:46289
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250624 14:18:31.268100 2974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:31.268719 2974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:18:31.269200 2974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:31.301558 2974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 14:18:31.301941 2974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:18:31.302207 2974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 14:18:31.302474 2974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 14:18:31.338418 2974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.2.14.62:46289
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:46289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:18:31.339861 2974 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:18:31.341570 2974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:18:31.358665 2981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:31.358739 2983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:31.359879 2974 server_base.cc:1048] running on GCE node
W20250624 14:18:31.359591 2980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:32.549312 2974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:18:32.552537 2974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:18:32.554052 2974 hybrid_clock.cc:648] HybridClock initialized: now 1750774712553997 us; error 67 us; skew 500 ppm
I20250624 14:18:32.554929 2974 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:18:32.561528 2974 webserver.cc:469] Webserver started at http://127.2.14.62:38395/ using document root <none> and password file <none>
I20250624 14:18:32.562525 2974 fs_manager.cc:362] Metadata directory not provided
I20250624 14:18:32.562736 2974 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:18:32.563247 2974 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:18:32.567920 2974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b5d2d831558741488d2b4c2d00af0a3a"
format_stamp: "Formatted at 2025-06-24 14:18:32 on dist-test-slave-hs17"
I20250624 14:18:32.569083 2974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b5d2d831558741488d2b4c2d00af0a3a"
format_stamp: "Formatted at 2025-06-24 14:18:32 on dist-test-slave-hs17"
I20250624 14:18:32.576547 2974 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250624 14:18:32.582340 2992 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:32.583505 2974 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250624 14:18:32.583935 2974 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b5d2d831558741488d2b4c2d00af0a3a"
format_stamp: "Formatted at 2025-06-24 14:18:32 on dist-test-slave-hs17"
I20250624 14:18:32.584290 2974 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:18:32.655676 2974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:18:32.657249 2974 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:18:32.657722 2974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:18:32.730247 2974 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.62:46289
I20250624 14:18:32.730326 3044 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.62:46289 every 8 connection(s)
I20250624 14:18:32.733094 2974 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 14:18:32.734835 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 2974
I20250624 14:18:32.735373 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 14:18:32.739715 3045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:32.763610 3045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a: Bootstrap starting.
I20250624 14:18:32.769236 3045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:32.770970 3045 log.cc:826] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a: Log is configured to *not* fsync() on all Append() calls
I20250624 14:18:32.775540 3045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a: No bootstrap required, opened a new log
I20250624 14:18:32.792994 3045 raft_consensus.cc:357] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d2d831558741488d2b4c2d00af0a3a" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 46289 } }
I20250624 14:18:32.793700 3045 raft_consensus.cc:383] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:18:32.793982 3045 raft_consensus.cc:738] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d2d831558741488d2b4c2d00af0a3a, State: Initialized, Role: FOLLOWER
I20250624 14:18:32.794699 3045 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d2d831558741488d2b4c2d00af0a3a" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 46289 } }
I20250624 14:18:32.795199 3045 raft_consensus.cc:397] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:18:32.795480 3045 raft_consensus.cc:491] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:18:32.795804 3045 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:18:32.800276 3045 raft_consensus.cc:513] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d2d831558741488d2b4c2d00af0a3a" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 46289 } }
I20250624 14:18:32.801072 3045 leader_election.cc:304] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5d2d831558741488d2b4c2d00af0a3a; no voters:
I20250624 14:18:32.802728 3045 leader_election.cc:290] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 14:18:32.803438 3050 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:18:32.805639 3050 raft_consensus.cc:695] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [term 1 LEADER]: Becoming Leader. State: Replica: b5d2d831558741488d2b4c2d00af0a3a, State: Running, Role: LEADER
I20250624 14:18:32.806326 3050 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d2d831558741488d2b4c2d00af0a3a" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 46289 } }
I20250624 14:18:32.806771 3045 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [sys.catalog]: configured and running, proceeding with master startup.
I20250624 14:18:32.816541 3052 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5d2d831558741488d2b4c2d00af0a3a. Latest consensus state: current_term: 1 leader_uuid: "b5d2d831558741488d2b4c2d00af0a3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d2d831558741488d2b4c2d00af0a3a" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 46289 } } }
I20250624 14:18:32.817442 3052 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [sys.catalog]: This master's current role is: LEADER
I20250624 14:18:32.818471 3051 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5d2d831558741488d2b4c2d00af0a3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d2d831558741488d2b4c2d00af0a3a" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 46289 } } }
I20250624 14:18:32.819226 3051 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a [sys.catalog]: This master's current role is: LEADER
I20250624 14:18:32.821209 3059 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 14:18:32.833312 3059 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 14:18:32.850541 3059 catalog_manager.cc:1349] Generated new cluster ID: 351588d6c2b043c5be9d3eedaaae6ae4
I20250624 14:18:32.850873 3059 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 14:18:32.870913 3059 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 14:18:32.872467 3059 catalog_manager.cc:1506] Loading token signing keys...
I20250624 14:18:32.891999 3059 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b5d2d831558741488d2b4c2d00af0a3a: Generated new TSK 0
I20250624 14:18:32.892989 3059 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 14:18:32.904069 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.1:0
--local_ip_for_outbound_sockets=127.2.14.1
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:46289
--builtin_ntp_servers=127.2.14.20:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250624 14:18:33.219795 3070 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250624 14:18:33.220454 3070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:33.220708 3070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:18:33.221208 3070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:33.252833 3070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:18:33.253733 3070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.1
I20250624 14:18:33.289677 3070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:46289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.1
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:18:33.291075 3070 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:18:33.292786 3070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:18:33.311316 3079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:33.315689 3076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:33.311417 3077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:34.808391 3078 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1494 milliseconds
I20250624 14:18:34.808496 3070 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:18:34.809835 3070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:18:34.812690 3070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:18:34.814392 3070 hybrid_clock.cc:648] HybridClock initialized: now 1750774714814338 us; error 70 us; skew 500 ppm
I20250624 14:18:34.815243 3070 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:18:34.823539 3070 webserver.cc:469] Webserver started at http://127.2.14.1:37197/ using document root <none> and password file <none>
I20250624 14:18:34.824587 3070 fs_manager.cc:362] Metadata directory not provided
I20250624 14:18:34.824795 3070 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:18:34.825311 3070 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:18:34.830132 3070 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7b579784b5984a29a25dbce3f55cfade"
format_stamp: "Formatted at 2025-06-24 14:18:34 on dist-test-slave-hs17"
I20250624 14:18:34.831303 3070 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7b579784b5984a29a25dbce3f55cfade"
format_stamp: "Formatted at 2025-06-24 14:18:34 on dist-test-slave-hs17"
I20250624 14:18:34.839288 3070 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250624 14:18:34.847110 3086 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:34.848450 3070 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250624 14:18:34.848866 3070 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7b579784b5984a29a25dbce3f55cfade"
format_stamp: "Formatted at 2025-06-24 14:18:34 on dist-test-slave-hs17"
I20250624 14:18:34.849323 3070 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:18:34.941071 3070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:18:34.942606 3070 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:18:34.943074 3070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:18:34.946193 3070 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:18:34.950599 3070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:18:34.950820 3070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:34.951112 3070 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:18:34.951264 3070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:35.136294 3070 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.1:37407
I20250624 14:18:35.136430 3198 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.1:37407 every 8 connection(s)
I20250624 14:18:35.139153 3070 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 14:18:35.147051 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 3070
I20250624 14:18:35.147468 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 14:18:35.153708 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.2:0
--local_ip_for_outbound_sockets=127.2.14.2
--webserver_interface=127.2.14.2
--webserver_port=0
--tserver_master_addrs=127.2.14.62:46289
--builtin_ntp_servers=127.2.14.20:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250624 14:18:35.168121 3199 heartbeater.cc:344] Connected to a master server at 127.2.14.62:46289
I20250624 14:18:35.168690 3199 heartbeater.cc:461] Registering TS with master...
I20250624 14:18:35.170128 3199 heartbeater.cc:507] Master 127.2.14.62:46289 requested a full tablet report, sending...
I20250624 14:18:35.173594 3009 ts_manager.cc:194] Registered new tserver with Master: 7b579784b5984a29a25dbce3f55cfade (127.2.14.1:37407)
I20250624 14:18:35.176548 3009 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.1:40805
W20250624 14:18:35.474956 3203 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250624 14:18:35.475641 3203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:35.475919 3203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:18:35.476406 3203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:35.507576 3203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:18:35.508464 3203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.2
I20250624 14:18:35.543923 3203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.14.2
--webserver_port=0
--tserver_master_addrs=127.2.14.62:46289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.2
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:18:35.545245 3203 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:18:35.546886 3203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:18:35.561836 3210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:36.182129 3199 heartbeater.cc:499] Master 127.2.14.62:46289 was elected leader, sending a full tablet report...
W20250624 14:18:35.562428 3209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:35.565064 3203 server_base.cc:1048] running on GCE node
W20250624 14:18:35.564046 3212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:36.759295 3203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:18:36.761622 3203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:18:36.763000 3203 hybrid_clock.cc:648] HybridClock initialized: now 1750774716762925 us; error 89 us; skew 500 ppm
I20250624 14:18:36.763870 3203 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:18:36.777635 3203 webserver.cc:469] Webserver started at http://127.2.14.2:39045/ using document root <none> and password file <none>
I20250624 14:18:36.778880 3203 fs_manager.cc:362] Metadata directory not provided
I20250624 14:18:36.779083 3203 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:18:36.779506 3203 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:18:36.784453 3203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9f6c80074df24b75925f768d83d26da6"
format_stamp: "Formatted at 2025-06-24 14:18:36 on dist-test-slave-hs17"
I20250624 14:18:36.785530 3203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9f6c80074df24b75925f768d83d26da6"
format_stamp: "Formatted at 2025-06-24 14:18:36 on dist-test-slave-hs17"
I20250624 14:18:36.792788 3203 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250624 14:18:36.798583 3219 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:36.799741 3203 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250624 14:18:36.800050 3203 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9f6c80074df24b75925f768d83d26da6"
format_stamp: "Formatted at 2025-06-24 14:18:36 on dist-test-slave-hs17"
I20250624 14:18:36.800364 3203 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:18:36.860210 3203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:18:36.861702 3203 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:18:36.862130 3203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:18:36.864706 3203 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:18:36.868865 3203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:18:36.869081 3203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:36.869287 3203 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:18:36.869419 3203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:37.008536 3203 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.2:44119
I20250624 14:18:37.008639 3331 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.2:44119 every 8 connection(s)
I20250624 14:18:37.011137 3203 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 14:18:37.018888 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 3203
I20250624 14:18:37.019496 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 14:18:37.025488 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.3:0
--local_ip_for_outbound_sockets=127.2.14.3
--webserver_interface=127.2.14.3
--webserver_port=0
--tserver_master_addrs=127.2.14.62:46289
--builtin_ntp_servers=127.2.14.20:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250624 14:18:37.034014 3332 heartbeater.cc:344] Connected to a master server at 127.2.14.62:46289
I20250624 14:18:37.034608 3332 heartbeater.cc:461] Registering TS with master...
I20250624 14:18:37.036082 3332 heartbeater.cc:507] Master 127.2.14.62:46289 requested a full tablet report, sending...
I20250624 14:18:37.038758 3009 ts_manager.cc:194] Registered new tserver with Master: 9f6c80074df24b75925f768d83d26da6 (127.2.14.2:44119)
I20250624 14:18:37.040095 3009 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.2:46365
W20250624 14:18:37.330910 3336 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250624 14:18:37.331597 3336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:37.331884 3336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:18:37.332384 3336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:37.363579 3336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:18:37.364471 3336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.3
I20250624 14:18:37.399966 3336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.14.3
--webserver_port=0
--tserver_master_addrs=127.2.14.62:46289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.3
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:18:37.401297 3336 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:18:37.402959 3336 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:18:37.418255 3343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:37.584568 3195 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2da0 after lost signal to thread 3071
W20250624 14:18:37.586349 3195 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5880 after lost signal to thread 3198
I20250624 14:18:38.044255 3332 heartbeater.cc:499] Master 127.2.14.62:46289 was elected leader, sending a full tablet report...
W20250624 14:18:37.420105 3345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:37.418277 3342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:38.923811 3344 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1507 milliseconds
I20250624 14:18:38.923915 3336 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:18:38.928319 3336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:18:38.931092 3336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:18:38.932549 3336 hybrid_clock.cc:648] HybridClock initialized: now 1750774718932490 us; error 74 us; skew 500 ppm
I20250624 14:18:38.933391 3336 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:18:38.940689 3336 webserver.cc:469] Webserver started at http://127.2.14.3:43979/ using document root <none> and password file <none>
I20250624 14:18:38.942940 3336 fs_manager.cc:362] Metadata directory not provided
I20250624 14:18:38.943207 3336 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:18:38.943722 3336 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:18:38.948365 3336 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e3cf6c34cf0d49748923d5e1609eee95"
format_stamp: "Formatted at 2025-06-24 14:18:38 on dist-test-slave-hs17"
I20250624 14:18:38.949594 3336 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e3cf6c34cf0d49748923d5e1609eee95"
format_stamp: "Formatted at 2025-06-24 14:18:38 on dist-test-slave-hs17"
I20250624 14:18:38.957340 3336 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20250624 14:18:38.963863 3352 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:38.965224 3336 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20250624 14:18:38.965653 3336 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e3cf6c34cf0d49748923d5e1609eee95"
format_stamp: "Formatted at 2025-06-24 14:18:38 on dist-test-slave-hs17"
I20250624 14:18:38.966140 3336 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:18:39.029269 3336 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:18:39.030795 3336 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:18:39.031239 3336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:18:39.034039 3336 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:18:39.038993 3336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:18:39.039222 3336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:39.039467 3336 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:18:39.039677 3336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250624 14:18:39.220786 3336 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.3:35361
I20250624 14:18:39.220965 3464 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.3:35361 every 8 connection(s)
I20250624 14:18:39.223423 3336 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 14:18:39.232964 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 3336
I20250624 14:18:39.233465 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 14:18:39.249392 3465 heartbeater.cc:344] Connected to a master server at 127.2.14.62:46289
I20250624 14:18:39.250088 3465 heartbeater.cc:461] Registering TS with master...
I20250624 14:18:39.251458 3465 heartbeater.cc:507] Master 127.2.14.62:46289 requested a full tablet report, sending...
I20250624 14:18:39.253983 3008 ts_manager.cc:194] Registered new tserver with Master: e3cf6c34cf0d49748923d5e1609eee95 (127.2.14.3:35361)
I20250624 14:18:39.255450 3008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.3:51471
I20250624 14:18:39.270383 2104 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 14:18:39.317916 3008 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33308:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250624 14:18:39.338443 3008 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 14:18:39.397099 3267 tablet_service.cc:1468] Processing CreateTablet for tablet 0e06ec5e5a5c456a960d440db2964a0e (DEFAULT_TABLE table=TestTable [id=28e286d039e049f68b988283a2cf5970]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:18:39.398234 3400 tablet_service.cc:1468] Processing CreateTablet for tablet 0e06ec5e5a5c456a960d440db2964a0e (DEFAULT_TABLE table=TestTable [id=28e286d039e049f68b988283a2cf5970]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:18:39.399466 3267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e06ec5e5a5c456a960d440db2964a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:39.400282 3400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e06ec5e5a5c456a960d440db2964a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:39.417268 3134 tablet_service.cc:1468] Processing CreateTablet for tablet 0e06ec5e5a5c456a960d440db2964a0e (DEFAULT_TABLE table=TestTable [id=28e286d039e049f68b988283a2cf5970]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:18:39.422632 3134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e06ec5e5a5c456a960d440db2964a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:39.437175 3485 tablet_bootstrap.cc:492] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95: Bootstrap starting.
I20250624 14:18:39.444068 3484 tablet_bootstrap.cc:492] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6: Bootstrap starting.
I20250624 14:18:39.447722 3485 tablet_bootstrap.cc:654] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:39.452080 3485 log.cc:826] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95: Log is configured to *not* fsync() on all Append() calls
I20250624 14:18:39.456390 3484 tablet_bootstrap.cc:654] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:39.459311 3484 log.cc:826] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6: Log is configured to *not* fsync() on all Append() calls
I20250624 14:18:39.462172 3485 tablet_bootstrap.cc:492] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95: No bootstrap required, opened a new log
I20250624 14:18:39.462718 3485 ts_tablet_manager.cc:1397] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.014s
I20250624 14:18:39.469862 3487 tablet_bootstrap.cc:492] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade: Bootstrap starting.
I20250624 14:18:39.483300 3487 tablet_bootstrap.cc:654] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:39.487067 3484 tablet_bootstrap.cc:492] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6: No bootstrap required, opened a new log
I20250624 14:18:39.487614 3484 ts_tablet_manager.cc:1397] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.027s
I20250624 14:18:39.489188 3487 log.cc:826] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade: Log is configured to *not* fsync() on all Append() calls
I20250624 14:18:39.491881 3485 raft_consensus.cc:357] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6c80074df24b75925f768d83d26da6" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44119 } } peers { permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 } } peers { permanent_uuid: "7b579784b5984a29a25dbce3f55cfade" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 37407 } }
I20250624 14:18:39.492882 3485 raft_consensus.cc:738] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cf6c34cf0d49748923d5e1609eee95, State: Initialized, Role: FOLLOWER
I20250624 14:18:39.493867 3485 consensus_queue.cc:260] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6c80074df24b75925f768d83d26da6" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44119 } } peers { permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 } } peers { permanent_uuid: "7b579784b5984a29a25dbce3f55cfade" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 37407 } }
I20250624 14:18:39.509269 3485 ts_tablet_manager.cc:1428] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95: Time spent starting tablet: real 0.046s user 0.039s sys 0.007s
I20250624 14:18:39.509582 3465 heartbeater.cc:499] Master 127.2.14.62:46289 was elected leader, sending a full tablet report...
I20250624 14:18:39.514564 3487 tablet_bootstrap.cc:492] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade: No bootstrap required, opened a new log
I20250624 14:18:39.515154 3487 ts_tablet_manager.cc:1397] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade: Time spent bootstrapping tablet: real 0.046s user 0.019s sys 0.017s
I20250624 14:18:39.523852 3484 raft_consensus.cc:357] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6c80074df24b75925f768d83d26da6" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44119 } } peers { permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 } } peers { permanent_uuid: "7b579784b5984a29a25dbce3f55cfade" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 37407 } }
I20250624 14:18:39.524895 3484 raft_consensus.cc:738] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6c80074df24b75925f768d83d26da6, State: Initialized, Role: FOLLOWER
I20250624 14:18:39.525668 3484 consensus_queue.cc:260] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6c80074df24b75925f768d83d26da6" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44119 } } peers { permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 } } peers { permanent_uuid: "7b579784b5984a29a25dbce3f55cfade" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 37407 } }
I20250624 14:18:39.530174 3484 ts_tablet_manager.cc:1428] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6: Time spent starting tablet: real 0.042s user 0.030s sys 0.005s
I20250624 14:18:39.540351 2104 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 14:18:39.545089 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7b579784b5984a29a25dbce3f55cfade to finish bootstrapping
I20250624 14:18:39.549489 3487 raft_consensus.cc:357] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6c80074df24b75925f768d83d26da6" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44119 } } peers { permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 } } peers { permanent_uuid: "7b579784b5984a29a25dbce3f55cfade" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 37407 } }
I20250624 14:18:39.550331 3487 raft_consensus.cc:738] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b579784b5984a29a25dbce3f55cfade, State: Initialized, Role: FOLLOWER
I20250624 14:18:39.551193 3487 consensus_queue.cc:260] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6c80074df24b75925f768d83d26da6" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44119 } } peers { permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 } } peers { permanent_uuid: "7b579784b5984a29a25dbce3f55cfade" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 37407 } }
I20250624 14:18:39.557760 3487 ts_tablet_manager.cc:1428] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade: Time spent starting tablet: real 0.042s user 0.034s sys 0.000s
I20250624 14:18:39.563201 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9f6c80074df24b75925f768d83d26da6 to finish bootstrapping
I20250624 14:18:39.576581 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e3cf6c34cf0d49748923d5e1609eee95 to finish bootstrapping
I20250624 14:18:39.626319 3154 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0e06ec5e5a5c456a960d440db2964a0e"
dest_uuid: "7b579784b5984a29a25dbce3f55cfade"
from {username='slave'} at 127.0.0.1:36954
I20250624 14:18:39.626900 3154 raft_consensus.cc:491] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250624 14:18:39.627234 3154 raft_consensus.cc:3058] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:18:39.631748 3154 raft_consensus.cc:513] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6c80074df24b75925f768d83d26da6" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44119 } } peers { permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 } } peers { permanent_uuid: "7b579784b5984a29a25dbce3f55cfade" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 37407 } }
I20250624 14:18:39.634025 3154 leader_election.cc:290] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [CANDIDATE]: Term 1 election: Requested vote from peers 9f6c80074df24b75925f768d83d26da6 (127.2.14.2:44119), e3cf6c34cf0d49748923d5e1609eee95 (127.2.14.3:35361)
I20250624 14:18:39.648154 2104 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
W20250624 14:18:39.649852 3200 tablet.cc:2378] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:18:39.650866 3287 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e06ec5e5a5c456a960d440db2964a0e" candidate_uuid: "7b579784b5984a29a25dbce3f55cfade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f6c80074df24b75925f768d83d26da6"
I20250624 14:18:39.651737 3287 raft_consensus.cc:3058] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:18:39.651808 3420 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e06ec5e5a5c456a960d440db2964a0e" candidate_uuid: "7b579784b5984a29a25dbce3f55cfade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e3cf6c34cf0d49748923d5e1609eee95"
I20250624 14:18:39.652455 3420 raft_consensus.cc:3058] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:18:39.656852 3420 raft_consensus.cc:2466] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b579784b5984a29a25dbce3f55cfade in term 1.
I20250624 14:18:39.658000 3087 leader_election.cc:304] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b579784b5984a29a25dbce3f55cfade, e3cf6c34cf0d49748923d5e1609eee95; no voters:
I20250624 14:18:39.658814 3493 raft_consensus.cc:2802] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:18:39.659274 3287 raft_consensus.cc:2466] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b579784b5984a29a25dbce3f55cfade in term 1.
I20250624 14:18:39.661844 3493 raft_consensus.cc:695] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 1 LEADER]: Becoming Leader. State: Replica: 7b579784b5984a29a25dbce3f55cfade, State: Running, Role: LEADER
I20250624 14:18:39.662698 3493 consensus_queue.cc:237] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6c80074df24b75925f768d83d26da6" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44119 } } peers { permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 } } peers { permanent_uuid: "7b579784b5984a29a25dbce3f55cfade" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 37407 } }
I20250624 14:18:39.673744 3009 catalog_manager.cc:5582] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b579784b5984a29a25dbce3f55cfade (127.2.14.1). New cstate: current_term: 1 leader_uuid: "7b579784b5984a29a25dbce3f55cfade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6c80074df24b75925f768d83d26da6" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b579784b5984a29a25dbce3f55cfade" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 37407 } health_report { overall_health: HEALTHY } } }
W20250624 14:18:39.728853 3466 tablet.cc:2378] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:18:39.755215 2104 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
W20250624 14:18:39.768882 3333 tablet.cc:2378] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:18:39.961115 2104 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250624 14:18:40.059621 3493 consensus_queue.cc:1035] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 14:18:40.089349 3497 consensus_queue.cc:1035] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6c80074df24b75925f768d83d26da6" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 14:18:41.950621 3154 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0e06ec5e5a5c456a960d440db2964a0e"
dest_uuid: "7b579784b5984a29a25dbce3f55cfade"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36960
I20250624 14:18:41.951311 3154 raft_consensus.cc:604] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 1 LEADER]: Received request to transfer leadership
I20250624 14:18:42.004901 3513 raft_consensus.cc:991] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade: : Instructing follower 9f6c80074df24b75925f768d83d26da6 to start an election
I20250624 14:18:42.005252 3530 raft_consensus.cc:1079] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 1 LEADER]: Signalling peer 9f6c80074df24b75925f768d83d26da6 to start an election
I20250624 14:18:42.006680 3287 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0e06ec5e5a5c456a960d440db2964a0e"
dest_uuid: "9f6c80074df24b75925f768d83d26da6"
from {username='slave'} at 127.2.14.1:50097
I20250624 14:18:42.007198 3287 raft_consensus.cc:491] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250624 14:18:42.007459 3287 raft_consensus.cc:3058] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [term 1 FOLLOWER]: Advancing to term 2
I20250624 14:18:42.012312 3287 raft_consensus.cc:513] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6c80074df24b75925f768d83d26da6" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44119 } } peers { permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 } } peers { permanent_uuid: "7b579784b5984a29a25dbce3f55cfade" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 37407 } }
I20250624 14:18:42.014770 3287 leader_election.cc:290] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [CANDIDATE]: Term 2 election: Requested vote from peers e3cf6c34cf0d49748923d5e1609eee95 (127.2.14.3:35361), 7b579784b5984a29a25dbce3f55cfade (127.2.14.1:37407)
I20250624 14:18:42.027019 3420 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e06ec5e5a5c456a960d440db2964a0e" candidate_uuid: "9f6c80074df24b75925f768d83d26da6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e3cf6c34cf0d49748923d5e1609eee95"
I20250624 14:18:42.027630 3420 raft_consensus.cc:3058] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95 [term 1 FOLLOWER]: Advancing to term 2
I20250624 14:18:42.029592 3154 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e06ec5e5a5c456a960d440db2964a0e" candidate_uuid: "9f6c80074df24b75925f768d83d26da6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b579784b5984a29a25dbce3f55cfade"
I20250624 14:18:42.030253 3154 raft_consensus.cc:3053] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 1 LEADER]: Stepping down as leader of term 1
I20250624 14:18:42.030582 3154 raft_consensus.cc:738] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b579784b5984a29a25dbce3f55cfade, State: Running, Role: LEADER
I20250624 14:18:42.031234 3154 consensus_queue.cc:260] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6c80074df24b75925f768d83d26da6" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44119 } } peers { permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 } } peers { permanent_uuid: "7b579784b5984a29a25dbce3f55cfade" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 37407 } }
I20250624 14:18:42.032552 3154 raft_consensus.cc:3058] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 1 FOLLOWER]: Advancing to term 2
I20250624 14:18:42.034044 3420 raft_consensus.cc:2466] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f6c80074df24b75925f768d83d26da6 in term 2.
I20250624 14:18:42.035276 3220 leader_election.cc:304] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f6c80074df24b75925f768d83d26da6, e3cf6c34cf0d49748923d5e1609eee95; no voters:
I20250624 14:18:42.037477 3534 raft_consensus.cc:2802] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 14:18:42.037624 3154 raft_consensus.cc:2466] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f6c80074df24b75925f768d83d26da6 in term 2.
I20250624 14:18:42.039301 3534 raft_consensus.cc:695] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [term 2 LEADER]: Becoming Leader. State: Replica: 9f6c80074df24b75925f768d83d26da6, State: Running, Role: LEADER
I20250624 14:18:42.040179 3534 consensus_queue.cc:237] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6c80074df24b75925f768d83d26da6" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44119 } } peers { permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 } } peers { permanent_uuid: "7b579784b5984a29a25dbce3f55cfade" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 37407 } }
I20250624 14:18:42.048484 3009 catalog_manager.cc:5582] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 reported cstate change: term changed from 1 to 2, leader changed from 7b579784b5984a29a25dbce3f55cfade (127.2.14.1) to 9f6c80074df24b75925f768d83d26da6 (127.2.14.2). New cstate: current_term: 2 leader_uuid: "9f6c80074df24b75925f768d83d26da6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6c80074df24b75925f768d83d26da6" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 44119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b579784b5984a29a25dbce3f55cfade" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 37407 } health_report { overall_health: UNKNOWN } } }
I20250624 14:18:42.542398 3420 raft_consensus.cc:1273] T 0e06ec5e5a5c456a960d440db2964a0e P e3cf6c34cf0d49748923d5e1609eee95 [term 2 FOLLOWER]: Refusing update from remote peer 9f6c80074df24b75925f768d83d26da6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250624 14:18:42.543648 3534 consensus_queue.cc:1035] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cf6c34cf0d49748923d5e1609eee95" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250624 14:18:42.556872 3154 raft_consensus.cc:1273] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 2 FOLLOWER]: Refusing update from remote peer 9f6c80074df24b75925f768d83d26da6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250624 14:18:42.559410 3546 consensus_queue.cc:1035] T 0e06ec5e5a5c456a960d440db2964a0e P 9f6c80074df24b75925f768d83d26da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b579784b5984a29a25dbce3f55cfade" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 37407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250624 14:18:44.684615 3154 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0e06ec5e5a5c456a960d440db2964a0e"
dest_uuid: "7b579784b5984a29a25dbce3f55cfade"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46342
I20250624 14:18:44.685499 3154 raft_consensus.cc:604] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 2 FOLLOWER]: Received request to transfer leadership
I20250624 14:18:44.685945 3154 raft_consensus.cc:612] T 0e06ec5e5a5c456a960d440db2964a0e P 7b579784b5984a29a25dbce3f55cfade [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250624 14:18:45.722940 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 3070
I20250624 14:18:45.750363 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 3203
I20250624 14:18:45.776319 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 3336
I20250624 14:18:45.803926 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 2974
2025-06-24T14:18:45Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14952 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250624 14:18:45.864355 2104 test_util.cc:276] Using random seed: 27126661
I20250624 14:18:45.868613 2104 ts_itest-base.cc:115] Starting cluster with:
I20250624 14:18:45.868793 2104 ts_itest-base.cc:116] --------------
I20250624 14:18:45.868966 2104 ts_itest-base.cc:117] 3 tablet servers
I20250624 14:18:45.869112 2104 ts_itest-base.cc:118] 3 replicas per TS
I20250624 14:18:45.869256 2104 ts_itest-base.cc:119] --------------
2025-06-24T14:18:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T14:18:45Z Disabled control of system clock
I20250624 14:18:45.909409 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:41629
--webserver_interface=127.2.14.62
--webserver_port=0
--builtin_ntp_servers=127.2.14.20:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.14.62:41629 with env {}
W20250624 14:18:46.216768 3580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:46.217334 3580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:18:46.217752 3580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:46.249363 3580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 14:18:46.249670 3580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:18:46.249887 3580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 14:18:46.250082 3580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 14:18:46.286275 3580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.14.62:41629
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:41629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:18:46.287611 3580 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:18:46.289280 3580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:18:46.305858 3587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:46.306073 3586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:46.305845 3589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:47.486383 3588 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 14:18:47.486430 3580 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:18:47.490672 3580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:18:47.493872 3580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:18:47.495359 3580 hybrid_clock.cc:648] HybridClock initialized: now 1750774727495312 us; error 57 us; skew 500 ppm
I20250624 14:18:47.496263 3580 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:18:47.509003 3580 webserver.cc:469] Webserver started at http://127.2.14.62:33829/ using document root <none> and password file <none>
I20250624 14:18:47.510039 3580 fs_manager.cc:362] Metadata directory not provided
I20250624 14:18:47.510254 3580 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:18:47.510864 3580 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:18:47.515712 3580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1910729051ae497592bf48a465eda4aa"
format_stamp: "Formatted at 2025-06-24 14:18:47 on dist-test-slave-hs17"
I20250624 14:18:47.516907 3580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1910729051ae497592bf48a465eda4aa"
format_stamp: "Formatted at 2025-06-24 14:18:47 on dist-test-slave-hs17"
I20250624 14:18:47.524367 3580 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250624 14:18:47.530043 3596 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:47.531095 3580 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250624 14:18:47.531436 3580 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1910729051ae497592bf48a465eda4aa"
format_stamp: "Formatted at 2025-06-24 14:18:47 on dist-test-slave-hs17"
I20250624 14:18:47.531852 3580 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:18:47.584023 3580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:18:47.585587 3580 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:18:47.586064 3580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:18:47.656714 3580 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.62:41629
I20250624 14:18:47.656785 3647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.62:41629 every 8 connection(s)
I20250624 14:18:47.659469 3580 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 14:18:47.664723 3648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:47.668416 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 3580
I20250624 14:18:47.668931 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 14:18:47.688807 3648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa: Bootstrap starting.
I20250624 14:18:47.695308 3648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:47.697379 3648 log.cc:826] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa: Log is configured to *not* fsync() on all Append() calls
I20250624 14:18:47.701968 3648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa: No bootstrap required, opened a new log
I20250624 14:18:47.719287 3648 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1910729051ae497592bf48a465eda4aa" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41629 } }
I20250624 14:18:47.720116 3648 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:18:47.720456 3648 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1910729051ae497592bf48a465eda4aa, State: Initialized, Role: FOLLOWER
I20250624 14:18:47.721341 3648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1910729051ae497592bf48a465eda4aa" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41629 } }
I20250624 14:18:47.721904 3648 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:18:47.722154 3648 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:18:47.722409 3648 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:18:47.726540 3648 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1910729051ae497592bf48a465eda4aa" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41629 } }
I20250624 14:18:47.727247 3648 leader_election.cc:304] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1910729051ae497592bf48a465eda4aa; no voters:
I20250624 14:18:47.728972 3648 leader_election.cc:290] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 14:18:47.729686 3653 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:18:47.732081 3653 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [term 1 LEADER]: Becoming Leader. State: Replica: 1910729051ae497592bf48a465eda4aa, State: Running, Role: LEADER
I20250624 14:18:47.733274 3648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [sys.catalog]: configured and running, proceeding with master startup.
I20250624 14:18:47.732935 3653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1910729051ae497592bf48a465eda4aa" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41629 } }
I20250624 14:18:47.746063 3655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1910729051ae497592bf48a465eda4aa. Latest consensus state: current_term: 1 leader_uuid: "1910729051ae497592bf48a465eda4aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1910729051ae497592bf48a465eda4aa" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41629 } } }
I20250624 14:18:47.746740 3655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [sys.catalog]: This master's current role is: LEADER
I20250624 14:18:47.748209 3654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1910729051ae497592bf48a465eda4aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1910729051ae497592bf48a465eda4aa" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41629 } } }
I20250624 14:18:47.749028 3654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa [sys.catalog]: This master's current role is: LEADER
I20250624 14:18:47.749795 3663 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 14:18:47.763233 3663 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 14:18:47.780375 3663 catalog_manager.cc:1349] Generated new cluster ID: 788b843044ce4c60bb1165bc0344ad96
I20250624 14:18:47.780654 3663 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 14:18:47.814518 3663 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 14:18:47.816107 3663 catalog_manager.cc:1506] Loading token signing keys...
I20250624 14:18:47.830605 3663 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1910729051ae497592bf48a465eda4aa: Generated new TSK 0
I20250624 14:18:47.831832 3663 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 14:18:47.854045 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.1:0
--local_ip_for_outbound_sockets=127.2.14.1
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41629
--builtin_ntp_servers=127.2.14.20:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 14:18:48.168370 3672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:48.168960 3672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:18:48.169479 3672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:48.206722 3672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:18:48.207684 3672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.1
I20250624 14:18:48.243889 3672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.1
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:18:48.245330 3672 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:18:48.247177 3672 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:18:48.265623 3679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:48.269358 3672 server_base.cc:1048] running on GCE node
W20250624 14:18:48.268363 3678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:48.267297 3681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:49.471011 3672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:18:49.474354 3672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:18:49.475947 3672 hybrid_clock.cc:648] HybridClock initialized: now 1750774729475890 us; error 52 us; skew 500 ppm
I20250624 14:18:49.477035 3672 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:18:49.485064 3672 webserver.cc:469] Webserver started at http://127.2.14.1:42215/ using document root <none> and password file <none>
I20250624 14:18:49.486290 3672 fs_manager.cc:362] Metadata directory not provided
I20250624 14:18:49.486552 3672 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:18:49.487126 3672 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:18:49.493831 3672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a1583058e5e941c3b3d4a5bc36f14847"
format_stamp: "Formatted at 2025-06-24 14:18:49 on dist-test-slave-hs17"
I20250624 14:18:49.495353 3672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a1583058e5e941c3b3d4a5bc36f14847"
format_stamp: "Formatted at 2025-06-24 14:18:49 on dist-test-slave-hs17"
I20250624 14:18:49.504894 3672 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20250624 14:18:49.512358 3688 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:49.513736 3672 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250624 14:18:49.514124 3672 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a1583058e5e941c3b3d4a5bc36f14847"
format_stamp: "Formatted at 2025-06-24 14:18:49 on dist-test-slave-hs17"
I20250624 14:18:49.514560 3672 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:18:49.568107 3672 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:18:49.569587 3672 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:18:49.570030 3672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:18:49.572587 3672 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:18:49.576820 3672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:18:49.577030 3672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:49.577236 3672 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:18:49.577373 3672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:49.731357 3672 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.1:36679
I20250624 14:18:49.731549 3800 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.1:36679 every 8 connection(s)
I20250624 14:18:49.734022 3672 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 14:18:49.738090 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 3672
I20250624 14:18:49.738736 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 14:18:49.749588 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.2:0
--local_ip_for_outbound_sockets=127.2.14.2
--webserver_interface=127.2.14.2
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41629
--builtin_ntp_servers=127.2.14.20:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:18:49.778465 3801 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41629
I20250624 14:18:49.779079 3801 heartbeater.cc:461] Registering TS with master...
I20250624 14:18:49.780627 3801 heartbeater.cc:507] Master 127.2.14.62:41629 requested a full tablet report, sending...
I20250624 14:18:49.784161 3613 ts_manager.cc:194] Registered new tserver with Master: a1583058e5e941c3b3d4a5bc36f14847 (127.2.14.1:36679)
I20250624 14:18:49.786978 3613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.1:39577
W20250624 14:18:50.074155 3805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:50.074664 3805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:18:50.075273 3805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:50.108245 3805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:18:50.109081 3805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.2
I20250624 14:18:50.145679 3805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.14.2
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.2
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:18:50.146951 3805 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:18:50.148696 3805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:18:50.167611 3812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:50.792028 3801 heartbeater.cc:499] Master 127.2.14.62:41629 was elected leader, sending a full tablet report...
W20250624 14:18:50.173115 3811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:50.173265 3814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:50.175208 3805 server_base.cc:1048] running on GCE node
I20250624 14:18:51.369267 3805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:18:51.371631 3805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:18:51.373034 3805 hybrid_clock.cc:648] HybridClock initialized: now 1750774731372995 us; error 59 us; skew 500 ppm
I20250624 14:18:51.373878 3805 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:18:51.380537 3805 webserver.cc:469] Webserver started at http://127.2.14.2:35161/ using document root <none> and password file <none>
I20250624 14:18:51.381525 3805 fs_manager.cc:362] Metadata directory not provided
I20250624 14:18:51.381731 3805 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:18:51.382237 3805 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:18:51.386833 3805 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "788205b488b54049bcf41e9867dcb48e"
format_stamp: "Formatted at 2025-06-24 14:18:51 on dist-test-slave-hs17"
I20250624 14:18:51.388298 3805 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "788205b488b54049bcf41e9867dcb48e"
format_stamp: "Formatted at 2025-06-24 14:18:51 on dist-test-slave-hs17"
I20250624 14:18:51.396126 3805 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250624 14:18:51.402374 3821 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:51.403687 3805 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250624 14:18:51.404047 3805 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "788205b488b54049bcf41e9867dcb48e"
format_stamp: "Formatted at 2025-06-24 14:18:51 on dist-test-slave-hs17"
I20250624 14:18:51.404383 3805 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:18:51.456930 3805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:18:51.458454 3805 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:18:51.458923 3805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:18:51.461661 3805 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:18:51.466789 3805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:18:51.467072 3805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:51.467365 3805 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:18:51.467592 3805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:51.604394 3805 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.2:45737
I20250624 14:18:51.604504 3933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.2:45737 every 8 connection(s)
I20250624 14:18:51.607056 3805 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 14:18:51.616396 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 3805
I20250624 14:18:51.616807 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 14:18:51.622504 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.3:0
--local_ip_for_outbound_sockets=127.2.14.3
--webserver_interface=127.2.14.3
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41629
--builtin_ntp_servers=127.2.14.20:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:18:51.628544 3934 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41629
I20250624 14:18:51.628995 3934 heartbeater.cc:461] Registering TS with master...
I20250624 14:18:51.630024 3934 heartbeater.cc:507] Master 127.2.14.62:41629 requested a full tablet report, sending...
I20250624 14:18:51.632314 3613 ts_manager.cc:194] Registered new tserver with Master: 788205b488b54049bcf41e9867dcb48e (127.2.14.2:45737)
I20250624 14:18:51.633548 3613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.2:51133
W20250624 14:18:51.937635 3938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:51.938177 3938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:18:51.938694 3938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:51.972898 3938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:18:51.973778 3938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.3
I20250624 14:18:52.009128 3938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.14.3
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.3
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:18:52.010515 3938 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:18:52.012249 3938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:18:52.029696 3946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:52.636734 3934 heartbeater.cc:499] Master 127.2.14.62:41629 was elected leader, sending a full tablet report...
W20250624 14:18:52.029729 3945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:18:52.031119 3948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:18:52.032078 3938 server_base.cc:1048] running on GCE node
I20250624 14:18:53.202775 3938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:18:53.205391 3938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:18:53.206775 3938 hybrid_clock.cc:648] HybridClock initialized: now 1750774733206736 us; error 53 us; skew 500 ppm
I20250624 14:18:53.207657 3938 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:18:53.220175 3938 webserver.cc:469] Webserver started at http://127.2.14.3:38829/ using document root <none> and password file <none>
I20250624 14:18:53.221175 3938 fs_manager.cc:362] Metadata directory not provided
I20250624 14:18:53.221383 3938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:18:53.221870 3938 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:18:53.226578 3938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b90de089fc1a4184ba39ef3bccbba83f"
format_stamp: "Formatted at 2025-06-24 14:18:53 on dist-test-slave-hs17"
I20250624 14:18:53.227782 3938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b90de089fc1a4184ba39ef3bccbba83f"
format_stamp: "Formatted at 2025-06-24 14:18:53 on dist-test-slave-hs17"
I20250624 14:18:53.235124 3938 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20250624 14:18:53.241039 3955 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:53.242123 3938 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250624 14:18:53.242456 3938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b90de089fc1a4184ba39ef3bccbba83f"
format_stamp: "Formatted at 2025-06-24 14:18:53 on dist-test-slave-hs17"
I20250624 14:18:53.242794 3938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:18:53.299378 3938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:18:53.300954 3938 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:18:53.301415 3938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:18:53.304032 3938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:18:53.308218 3938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:18:53.308447 3938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:53.308715 3938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:18:53.308864 3938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:18:53.445926 3938 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.3:42561
I20250624 14:18:53.446022 4067 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.3:42561 every 8 connection(s)
I20250624 14:18:53.448573 3938 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 14:18:53.451037 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 3938
I20250624 14:18:53.451802 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 14:18:53.470541 4068 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41629
I20250624 14:18:53.471058 4068 heartbeater.cc:461] Registering TS with master...
I20250624 14:18:53.472326 4068 heartbeater.cc:507] Master 127.2.14.62:41629 requested a full tablet report, sending...
I20250624 14:18:53.474447 3613 ts_manager.cc:194] Registered new tserver with Master: b90de089fc1a4184ba39ef3bccbba83f (127.2.14.3:42561)
I20250624 14:18:53.475776 3613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.3:39163
I20250624 14:18:53.487869 2104 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 14:18:53.524601 3613 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53304:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250624 14:18:53.544279 3613 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 14:18:53.604207 4003 tablet_service.cc:1468] Processing CreateTablet for tablet d8695b54237a40dba758791933820b11 (DEFAULT_TABLE table=TestTable [id=ade2e4cac4e2400094b4bd650b2084ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:18:53.606835 4003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8695b54237a40dba758791933820b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:53.608762 3869 tablet_service.cc:1468] Processing CreateTablet for tablet d8695b54237a40dba758791933820b11 (DEFAULT_TABLE table=TestTable [id=ade2e4cac4e2400094b4bd650b2084ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:18:53.611301 3869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8695b54237a40dba758791933820b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:53.617034 3736 tablet_service.cc:1468] Processing CreateTablet for tablet d8695b54237a40dba758791933820b11 (DEFAULT_TABLE table=TestTable [id=ade2e4cac4e2400094b4bd650b2084ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:18:53.619231 3736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8695b54237a40dba758791933820b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:53.638800 4087 tablet_bootstrap.cc:492] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f: Bootstrap starting.
I20250624 14:18:53.646167 4088 tablet_bootstrap.cc:492] T d8695b54237a40dba758791933820b11 P 788205b488b54049bcf41e9867dcb48e: Bootstrap starting.
I20250624 14:18:53.646644 4087 tablet_bootstrap.cc:654] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:53.649272 4087 log.cc:826] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f: Log is configured to *not* fsync() on all Append() calls
I20250624 14:18:53.656229 4088 tablet_bootstrap.cc:654] T d8695b54237a40dba758791933820b11 P 788205b488b54049bcf41e9867dcb48e: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:53.659600 4088 log.cc:826] T d8695b54237a40dba758791933820b11 P 788205b488b54049bcf41e9867dcb48e: Log is configured to *not* fsync() on all Append() calls
I20250624 14:18:53.661140 4087 tablet_bootstrap.cc:492] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f: No bootstrap required, opened a new log
I20250624 14:18:53.661736 4087 ts_tablet_manager.cc:1397] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.011s
I20250624 14:18:53.669363 4090 tablet_bootstrap.cc:492] T d8695b54237a40dba758791933820b11 P a1583058e5e941c3b3d4a5bc36f14847: Bootstrap starting.
I20250624 14:18:53.671965 4088 tablet_bootstrap.cc:492] T d8695b54237a40dba758791933820b11 P 788205b488b54049bcf41e9867dcb48e: No bootstrap required, opened a new log
I20250624 14:18:53.672638 4088 ts_tablet_manager.cc:1397] T d8695b54237a40dba758791933820b11 P 788205b488b54049bcf41e9867dcb48e: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.007s
I20250624 14:18:53.677713 4090 tablet_bootstrap.cc:654] T d8695b54237a40dba758791933820b11 P a1583058e5e941c3b3d4a5bc36f14847: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:53.680691 4090 log.cc:826] T d8695b54237a40dba758791933820b11 P a1583058e5e941c3b3d4a5bc36f14847: Log is configured to *not* fsync() on all Append() calls
I20250624 14:18:53.690078 4087 raft_consensus.cc:357] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } } peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } }
I20250624 14:18:53.691270 4087 raft_consensus.cc:383] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:18:53.691681 4087 raft_consensus.cc:738] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90de089fc1a4184ba39ef3bccbba83f, State: Initialized, Role: FOLLOWER
I20250624 14:18:53.692773 4087 consensus_queue.cc:260] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } } peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } }
I20250624 14:18:53.701841 4068 heartbeater.cc:499] Master 127.2.14.62:41629 was elected leader, sending a full tablet report...
I20250624 14:18:53.704777 4087 ts_tablet_manager.cc:1428] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f: Time spent starting tablet: real 0.043s user 0.021s sys 0.017s
I20250624 14:18:53.707288 4088 raft_consensus.cc:357] T d8695b54237a40dba758791933820b11 P 788205b488b54049bcf41e9867dcb48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } } peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } }
I20250624 14:18:53.708401 4088 raft_consensus.cc:383] T d8695b54237a40dba758791933820b11 P 788205b488b54049bcf41e9867dcb48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:18:53.708774 4088 raft_consensus.cc:738] T d8695b54237a40dba758791933820b11 P 788205b488b54049bcf41e9867dcb48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788205b488b54049bcf41e9867dcb48e, State: Initialized, Role: FOLLOWER
I20250624 14:18:53.709192 4090 tablet_bootstrap.cc:492] T d8695b54237a40dba758791933820b11 P a1583058e5e941c3b3d4a5bc36f14847: No bootstrap required, opened a new log
I20250624 14:18:53.710060 4090 ts_tablet_manager.cc:1397] T d8695b54237a40dba758791933820b11 P a1583058e5e941c3b3d4a5bc36f14847: Time spent bootstrapping tablet: real 0.041s user 0.022s sys 0.017s
I20250624 14:18:53.709786 4088 consensus_queue.cc:260] T d8695b54237a40dba758791933820b11 P 788205b488b54049bcf41e9867dcb48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } } peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } }
I20250624 14:18:53.716646 4088 ts_tablet_manager.cc:1428] T d8695b54237a40dba758791933820b11 P 788205b488b54049bcf41e9867dcb48e: Time spent starting tablet: real 0.044s user 0.030s sys 0.012s
I20250624 14:18:53.742444 4090 raft_consensus.cc:357] T d8695b54237a40dba758791933820b11 P a1583058e5e941c3b3d4a5bc36f14847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } } peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } }
I20250624 14:18:53.743394 4090 raft_consensus.cc:383] T d8695b54237a40dba758791933820b11 P a1583058e5e941c3b3d4a5bc36f14847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:18:53.743709 4090 raft_consensus.cc:738] T d8695b54237a40dba758791933820b11 P a1583058e5e941c3b3d4a5bc36f14847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1583058e5e941c3b3d4a5bc36f14847, State: Initialized, Role: FOLLOWER
I20250624 14:18:53.744596 4090 consensus_queue.cc:260] T d8695b54237a40dba758791933820b11 P a1583058e5e941c3b3d4a5bc36f14847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } } peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } }
I20250624 14:18:53.748680 4090 ts_tablet_manager.cc:1428] T d8695b54237a40dba758791933820b11 P a1583058e5e941c3b3d4a5bc36f14847: Time spent starting tablet: real 0.038s user 0.024s sys 0.008s
W20250624 14:18:53.761559 3802 tablet.cc:2378] T d8695b54237a40dba758791933820b11 P a1583058e5e941c3b3d4a5bc36f14847: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:18:53.837304 4093 raft_consensus.cc:491] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 14:18:53.837800 4093 raft_consensus.cc:513] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } } peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } }
I20250624 14:18:53.840303 4093 leader_election.cc:290] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1583058e5e941c3b3d4a5bc36f14847 (127.2.14.1:36679), 788205b488b54049bcf41e9867dcb48e (127.2.14.2:45737)
I20250624 14:18:53.850149 3756 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8695b54237a40dba758791933820b11" candidate_uuid: "b90de089fc1a4184ba39ef3bccbba83f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1583058e5e941c3b3d4a5bc36f14847" is_pre_election: true
I20250624 14:18:53.851130 3756 raft_consensus.cc:2466] T d8695b54237a40dba758791933820b11 P a1583058e5e941c3b3d4a5bc36f14847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b90de089fc1a4184ba39ef3bccbba83f in term 0.
I20250624 14:18:53.852101 3889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8695b54237a40dba758791933820b11" candidate_uuid: "b90de089fc1a4184ba39ef3bccbba83f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "788205b488b54049bcf41e9867dcb48e" is_pre_election: true
I20250624 14:18:53.852562 3957 leader_election.cc:304] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1583058e5e941c3b3d4a5bc36f14847, b90de089fc1a4184ba39ef3bccbba83f; no voters:
I20250624 14:18:53.852974 3889 raft_consensus.cc:2466] T d8695b54237a40dba758791933820b11 P 788205b488b54049bcf41e9867dcb48e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b90de089fc1a4184ba39ef3bccbba83f in term 0.
I20250624 14:18:53.853441 4093 raft_consensus.cc:2802] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 14:18:53.853729 4093 raft_consensus.cc:491] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 14:18:53.854044 4093 raft_consensus.cc:3058] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:18:53.858716 4093 raft_consensus.cc:513] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } } peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } }
I20250624 14:18:53.860153 4093 leader_election.cc:290] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [CANDIDATE]: Term 1 election: Requested vote from peers a1583058e5e941c3b3d4a5bc36f14847 (127.2.14.1:36679), 788205b488b54049bcf41e9867dcb48e (127.2.14.2:45737)
I20250624 14:18:53.861029 3756 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8695b54237a40dba758791933820b11" candidate_uuid: "b90de089fc1a4184ba39ef3bccbba83f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1583058e5e941c3b3d4a5bc36f14847"
I20250624 14:18:53.861066 3889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8695b54237a40dba758791933820b11" candidate_uuid: "b90de089fc1a4184ba39ef3bccbba83f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "788205b488b54049bcf41e9867dcb48e"
I20250624 14:18:53.861454 3756 raft_consensus.cc:3058] T d8695b54237a40dba758791933820b11 P a1583058e5e941c3b3d4a5bc36f14847 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:18:53.861479 3889 raft_consensus.cc:3058] T d8695b54237a40dba758791933820b11 P 788205b488b54049bcf41e9867dcb48e [term 0 FOLLOWER]: Advancing to term 1
W20250624 14:18:53.864140 3935 tablet.cc:2378] T d8695b54237a40dba758791933820b11 P 788205b488b54049bcf41e9867dcb48e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:18:53.866631 3756 raft_consensus.cc:2466] T d8695b54237a40dba758791933820b11 P a1583058e5e941c3b3d4a5bc36f14847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b90de089fc1a4184ba39ef3bccbba83f in term 1.
I20250624 14:18:53.866639 3889 raft_consensus.cc:2466] T d8695b54237a40dba758791933820b11 P 788205b488b54049bcf41e9867dcb48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b90de089fc1a4184ba39ef3bccbba83f in term 1.
I20250624 14:18:53.867640 3957 leader_election.cc:304] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1583058e5e941c3b3d4a5bc36f14847, b90de089fc1a4184ba39ef3bccbba83f; no voters:
I20250624 14:18:53.868258 4093 raft_consensus.cc:2802] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:18:53.869901 4093 raft_consensus.cc:695] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [term 1 LEADER]: Becoming Leader. State: Replica: b90de089fc1a4184ba39ef3bccbba83f, State: Running, Role: LEADER
I20250624 14:18:53.870748 4093 consensus_queue.cc:237] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } } peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } }
I20250624 14:18:53.881784 3611 catalog_manager.cc:5582] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f reported cstate change: term changed from 0 to 1, leader changed from <none> to b90de089fc1a4184ba39ef3bccbba83f (127.2.14.3). New cstate: current_term: 1 leader_uuid: "b90de089fc1a4184ba39ef3bccbba83f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } health_report { overall_health: UNKNOWN } } }
I20250624 14:18:53.904471 2104 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 14:18:53.907990 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a1583058e5e941c3b3d4a5bc36f14847 to finish bootstrapping
I20250624 14:18:53.920850 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 788205b488b54049bcf41e9867dcb48e to finish bootstrapping
I20250624 14:18:53.932178 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b90de089fc1a4184ba39ef3bccbba83f to finish bootstrapping
I20250624 14:18:53.946353 3611 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53304:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250624 14:18:53.948019 3611 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250624 14:18:53.953802 4069 tablet.cc:2378] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:18:53.965039 3736 tablet_service.cc:1468] Processing CreateTablet for tablet 8eaa904bab134ea2856fe7ca743d1439 (DEFAULT_TABLE table=TestAnotherTable [id=cdf70cc94a9b476fb7cbc908f664fe9f]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250624 14:18:53.965713 3869 tablet_service.cc:1468] Processing CreateTablet for tablet 8eaa904bab134ea2856fe7ca743d1439 (DEFAULT_TABLE table=TestAnotherTable [id=cdf70cc94a9b476fb7cbc908f664fe9f]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250624 14:18:53.966209 3736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eaa904bab134ea2856fe7ca743d1439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:53.966826 3869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eaa904bab134ea2856fe7ca743d1439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:53.966564 4003 tablet_service.cc:1468] Processing CreateTablet for tablet 8eaa904bab134ea2856fe7ca743d1439 (DEFAULT_TABLE table=TestAnotherTable [id=cdf70cc94a9b476fb7cbc908f664fe9f]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250624 14:18:53.967730 4003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eaa904bab134ea2856fe7ca743d1439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:18:53.978096 4090 tablet_bootstrap.cc:492] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847: Bootstrap starting.
I20250624 14:18:53.982200 4088 tablet_bootstrap.cc:492] T 8eaa904bab134ea2856fe7ca743d1439 P 788205b488b54049bcf41e9867dcb48e: Bootstrap starting.
I20250624 14:18:53.982188 4087 tablet_bootstrap.cc:492] T 8eaa904bab134ea2856fe7ca743d1439 P b90de089fc1a4184ba39ef3bccbba83f: Bootstrap starting.
I20250624 14:18:53.984683 4090 tablet_bootstrap.cc:654] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:53.988261 4088 tablet_bootstrap.cc:654] T 8eaa904bab134ea2856fe7ca743d1439 P 788205b488b54049bcf41e9867dcb48e: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:53.988572 4087 tablet_bootstrap.cc:654] T 8eaa904bab134ea2856fe7ca743d1439 P b90de089fc1a4184ba39ef3bccbba83f: Neither blocks nor log segments found. Creating new log.
I20250624 14:18:53.991000 4090 tablet_bootstrap.cc:492] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847: No bootstrap required, opened a new log
I20250624 14:18:53.991605 4090 ts_tablet_manager.cc:1397] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20250624 14:18:53.994197 4088 tablet_bootstrap.cc:492] T 8eaa904bab134ea2856fe7ca743d1439 P 788205b488b54049bcf41e9867dcb48e: No bootstrap required, opened a new log
I20250624 14:18:53.994678 4088 ts_tablet_manager.cc:1397] T 8eaa904bab134ea2856fe7ca743d1439 P 788205b488b54049bcf41e9867dcb48e: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20250624 14:18:53.994429 4090 raft_consensus.cc:357] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } } peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } }
I20250624 14:18:53.995007 4087 tablet_bootstrap.cc:492] T 8eaa904bab134ea2856fe7ca743d1439 P b90de089fc1a4184ba39ef3bccbba83f: No bootstrap required, opened a new log
I20250624 14:18:53.995157 4090 raft_consensus.cc:383] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:18:53.995461 4087 ts_tablet_manager.cc:1397] T 8eaa904bab134ea2856fe7ca743d1439 P b90de089fc1a4184ba39ef3bccbba83f: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250624 14:18:53.995491 4090 raft_consensus.cc:738] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1583058e5e941c3b3d4a5bc36f14847, State: Initialized, Role: FOLLOWER
I20250624 14:18:53.996296 4090 consensus_queue.cc:260] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } } peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } }
I20250624 14:18:53.997483 4088 raft_consensus.cc:357] T 8eaa904bab134ea2856fe7ca743d1439 P 788205b488b54049bcf41e9867dcb48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } } peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } }
I20250624 14:18:53.997624 4087 raft_consensus.cc:357] T 8eaa904bab134ea2856fe7ca743d1439 P b90de089fc1a4184ba39ef3bccbba83f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } } peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } }
I20250624 14:18:53.998245 4088 raft_consensus.cc:383] T 8eaa904bab134ea2856fe7ca743d1439 P 788205b488b54049bcf41e9867dcb48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:18:53.998348 4087 raft_consensus.cc:383] T 8eaa904bab134ea2856fe7ca743d1439 P b90de089fc1a4184ba39ef3bccbba83f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:18:53.998509 4090 ts_tablet_manager.cc:1428] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250624 14:18:53.998636 4088 raft_consensus.cc:738] T 8eaa904bab134ea2856fe7ca743d1439 P 788205b488b54049bcf41e9867dcb48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788205b488b54049bcf41e9867dcb48e, State: Initialized, Role: FOLLOWER
I20250624 14:18:53.998660 4087 raft_consensus.cc:738] T 8eaa904bab134ea2856fe7ca743d1439 P b90de089fc1a4184ba39ef3bccbba83f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90de089fc1a4184ba39ef3bccbba83f, State: Initialized, Role: FOLLOWER
I20250624 14:18:53.999320 4088 consensus_queue.cc:260] T 8eaa904bab134ea2856fe7ca743d1439 P 788205b488b54049bcf41e9867dcb48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } } peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } }
I20250624 14:18:53.999382 4087 consensus_queue.cc:260] T 8eaa904bab134ea2856fe7ca743d1439 P b90de089fc1a4184ba39ef3bccbba83f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } } peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } }
I20250624 14:18:54.001555 4087 ts_tablet_manager.cc:1428] T 8eaa904bab134ea2856fe7ca743d1439 P b90de089fc1a4184ba39ef3bccbba83f: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20250624 14:18:54.004468 4088 ts_tablet_manager.cc:1428] T 8eaa904bab134ea2856fe7ca743d1439 P 788205b488b54049bcf41e9867dcb48e: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20250624 14:18:54.211019 4095 raft_consensus.cc:491] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 14:18:54.211431 4095 raft_consensus.cc:513] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } } peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } }
I20250624 14:18:54.213783 4095 leader_election.cc:290] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 788205b488b54049bcf41e9867dcb48e (127.2.14.2:45737), b90de089fc1a4184ba39ef3bccbba83f (127.2.14.3:42561)
I20250624 14:18:54.225044 3889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8eaa904bab134ea2856fe7ca743d1439" candidate_uuid: "a1583058e5e941c3b3d4a5bc36f14847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "788205b488b54049bcf41e9867dcb48e" is_pre_election: true
I20250624 14:18:54.225261 4023 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8eaa904bab134ea2856fe7ca743d1439" candidate_uuid: "a1583058e5e941c3b3d4a5bc36f14847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90de089fc1a4184ba39ef3bccbba83f" is_pre_election: true
I20250624 14:18:54.225543 3889 raft_consensus.cc:2466] T 8eaa904bab134ea2856fe7ca743d1439 P 788205b488b54049bcf41e9867dcb48e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1583058e5e941c3b3d4a5bc36f14847 in term 0.
I20250624 14:18:54.225919 4023 raft_consensus.cc:2466] T 8eaa904bab134ea2856fe7ca743d1439 P b90de089fc1a4184ba39ef3bccbba83f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1583058e5e941c3b3d4a5bc36f14847 in term 0.
I20250624 14:18:54.226544 3692 leader_election.cc:304] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 788205b488b54049bcf41e9867dcb48e, a1583058e5e941c3b3d4a5bc36f14847; no voters:
I20250624 14:18:54.227334 4095 raft_consensus.cc:2802] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 14:18:54.227700 4095 raft_consensus.cc:491] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 14:18:54.227942 4095 raft_consensus.cc:3058] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:18:54.232244 4095 raft_consensus.cc:513] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } } peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } }
I20250624 14:18:54.233718 4095 leader_election.cc:290] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [CANDIDATE]: Term 1 election: Requested vote from peers 788205b488b54049bcf41e9867dcb48e (127.2.14.2:45737), b90de089fc1a4184ba39ef3bccbba83f (127.2.14.3:42561)
I20250624 14:18:54.234577 3889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8eaa904bab134ea2856fe7ca743d1439" candidate_uuid: "a1583058e5e941c3b3d4a5bc36f14847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "788205b488b54049bcf41e9867dcb48e"
I20250624 14:18:54.234761 4023 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8eaa904bab134ea2856fe7ca743d1439" candidate_uuid: "a1583058e5e941c3b3d4a5bc36f14847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90de089fc1a4184ba39ef3bccbba83f"
I20250624 14:18:54.235008 3889 raft_consensus.cc:3058] T 8eaa904bab134ea2856fe7ca743d1439 P 788205b488b54049bcf41e9867dcb48e [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:18:54.235212 4023 raft_consensus.cc:3058] T 8eaa904bab134ea2856fe7ca743d1439 P b90de089fc1a4184ba39ef3bccbba83f [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:18:54.239010 3889 raft_consensus.cc:2466] T 8eaa904bab134ea2856fe7ca743d1439 P 788205b488b54049bcf41e9867dcb48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1583058e5e941c3b3d4a5bc36f14847 in term 1.
I20250624 14:18:54.239593 4023 raft_consensus.cc:2466] T 8eaa904bab134ea2856fe7ca743d1439 P b90de089fc1a4184ba39ef3bccbba83f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1583058e5e941c3b3d4a5bc36f14847 in term 1.
I20250624 14:18:54.239929 3692 leader_election.cc:304] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 788205b488b54049bcf41e9867dcb48e, a1583058e5e941c3b3d4a5bc36f14847; no voters:
I20250624 14:18:54.240629 4095 raft_consensus.cc:2802] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:18:54.242209 4095 raft_consensus.cc:695] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [term 1 LEADER]: Becoming Leader. State: Replica: a1583058e5e941c3b3d4a5bc36f14847, State: Running, Role: LEADER
I20250624 14:18:54.242929 4095 consensus_queue.cc:237] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } } peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } }
I20250624 14:18:54.253932 3611 catalog_manager.cc:5582] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1583058e5e941c3b3d4a5bc36f14847 (127.2.14.1). New cstate: current_term: 1 leader_uuid: "a1583058e5e941c3b3d4a5bc36f14847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 } health_report { overall_health: UNKNOWN } } }
I20250624 14:18:54.262503 4093 consensus_queue.cc:1035] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [LEADER]: Connected to new peer: Peer: permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 14:18:54.278191 4093 consensus_queue.cc:1035] T d8695b54237a40dba758791933820b11 P b90de089fc1a4184ba39ef3bccbba83f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1583058e5e941c3b3d4a5bc36f14847" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250624 14:18:54.596266 4112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:54.596892 4112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:54.645711 4112 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250624 14:18:54.652608 4095 consensus_queue.cc:1035] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "788205b488b54049bcf41e9867dcb48e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 45737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 14:18:54.680886 4105 consensus_queue.cc:1035] T 8eaa904bab134ea2856fe7ca743d1439 P a1583058e5e941c3b3d4a5bc36f14847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90de089fc1a4184ba39ef3bccbba83f" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250624 14:18:55.989866 4112 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.284s user 0.000s sys 0.003s
W20250624 14:18:55.990177 4112 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.285s user 0.000s sys 0.003s
W20250624 14:18:57.384522 4131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:18:57.385110 4131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:18:57.416683 4131 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250624 14:18:58.772892 4131 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.314s user 0.525s sys 0.782s
W20250624 14:18:58.773195 4131 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.314s user 0.525s sys 0.782s
W20250624 14:19:00.182405 4148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:00.183064 4148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:00.221606 4148 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250624 14:19:01.537025 4148 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.256s user 0.473s sys 0.781s
W20250624 14:19:01.537345 4148 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.256s user 0.473s sys 0.781s
W20250624 14:19:02.926843 4166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:02.927433 4166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:02.959508 4166 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250624 14:19:04.258126 4166 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.255s user 0.447s sys 0.803s
W20250624 14:19:04.258424 4166 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.255s user 0.447s sys 0.803s
I20250624 14:19:05.336941 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 3672
I20250624 14:19:05.365494 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 3805
I20250624 14:19:05.393555 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 3938
I20250624 14:19:05.421808 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 3580
2025-06-24T14:19:05Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (19620 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250624 14:19:05.485528 2104 test_util.cc:276] Using random seed: 46747833
I20250624 14:19:05.489943 2104 ts_itest-base.cc:115] Starting cluster with:
I20250624 14:19:05.490118 2104 ts_itest-base.cc:116] --------------
I20250624 14:19:05.490235 2104 ts_itest-base.cc:117] 3 tablet servers
I20250624 14:19:05.490343 2104 ts_itest-base.cc:118] 3 replicas per TS
I20250624 14:19:05.490449 2104 ts_itest-base.cc:119] --------------
2025-06-24T14:19:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T14:19:05Z Disabled control of system clock
I20250624 14:19:05.531185 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:41831
--webserver_interface=127.2.14.62
--webserver_port=0
--builtin_ntp_servers=127.2.14.20:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.14.62:41831
--superuser_acl=no-such-user with env {}
W20250624 14:19:05.839282 4186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:05.839920 4186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:05.840378 4186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:05.871716 4186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 14:19:05.872025 4186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:05.872226 4186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 14:19:05.872421 4186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 14:19:05.908403 4186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.14.62:41831
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:41831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:05.909678 4186 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:05.911630 4186 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:05.926028 4192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:05.926590 4193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:05.928463 4186 server_base.cc:1048] running on GCE node
W20250624 14:19:05.928901 4195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:07.120023 4186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:07.123257 4186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:07.124948 4186 hybrid_clock.cc:648] HybridClock initialized: now 1750774747125002 us; error 173 us; skew 500 ppm
I20250624 14:19:07.126163 4186 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:07.134181 4186 webserver.cc:469] Webserver started at http://127.2.14.62:43987/ using document root <none> and password file <none>
I20250624 14:19:07.135243 4186 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:07.135517 4186 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:07.136121 4186 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:19:07.141059 4186 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "27ff67805a64429db27c904ac59feecb"
format_stamp: "Formatted at 2025-06-24 14:19:07 on dist-test-slave-hs17"
I20250624 14:19:07.142266 4186 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "27ff67805a64429db27c904ac59feecb"
format_stamp: "Formatted at 2025-06-24 14:19:07 on dist-test-slave-hs17"
I20250624 14:19:07.150149 4186 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250624 14:19:07.156169 4202 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:07.157303 4186 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250624 14:19:07.157651 4186 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
uuid: "27ff67805a64429db27c904ac59feecb"
format_stamp: "Formatted at 2025-06-24 14:19:07 on dist-test-slave-hs17"
I20250624 14:19:07.157999 4186 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:07.209453 4186 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:07.211035 4186 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:07.211503 4186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:07.283880 4186 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.62:41831
I20250624 14:19:07.283980 4253 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.62:41831 every 8 connection(s)
I20250624 14:19:07.286641 4186 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 14:19:07.289266 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 4186
I20250624 14:19:07.289793 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 14:19:07.293416 4254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:19:07.319118 4254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb: Bootstrap starting.
I20250624 14:19:07.324914 4254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb: Neither blocks nor log segments found. Creating new log.
I20250624 14:19:07.326999 4254 log.cc:826] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:07.332022 4254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb: No bootstrap required, opened a new log
I20250624 14:19:07.349807 4254 raft_consensus.cc:357] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ff67805a64429db27c904ac59feecb" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41831 } }
I20250624 14:19:07.350476 4254 raft_consensus.cc:383] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:19:07.350768 4254 raft_consensus.cc:738] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ff67805a64429db27c904ac59feecb, State: Initialized, Role: FOLLOWER
I20250624 14:19:07.351531 4254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ff67805a64429db27c904ac59feecb" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41831 } }
I20250624 14:19:07.352077 4254 raft_consensus.cc:397] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:19:07.352344 4254 raft_consensus.cc:491] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:19:07.352658 4254 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:19:07.357251 4254 raft_consensus.cc:513] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ff67805a64429db27c904ac59feecb" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41831 } }
I20250624 14:19:07.358089 4254 leader_election.cc:304] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27ff67805a64429db27c904ac59feecb; no voters:
I20250624 14:19:07.359892 4254 leader_election.cc:290] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 14:19:07.360667 4259 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:19:07.363442 4259 raft_consensus.cc:695] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [term 1 LEADER]: Becoming Leader. State: Replica: 27ff67805a64429db27c904ac59feecb, State: Running, Role: LEADER
I20250624 14:19:07.365010 4254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [sys.catalog]: configured and running, proceeding with master startup.
I20250624 14:19:07.364638 4259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ff67805a64429db27c904ac59feecb" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41831 } }
I20250624 14:19:07.376904 4260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27ff67805a64429db27c904ac59feecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ff67805a64429db27c904ac59feecb" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41831 } } }
I20250624 14:19:07.379355 4260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [sys.catalog]: This master's current role is: LEADER
I20250624 14:19:07.378437 4261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27ff67805a64429db27c904ac59feecb. Latest consensus state: current_term: 1 leader_uuid: "27ff67805a64429db27c904ac59feecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ff67805a64429db27c904ac59feecb" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41831 } } }
I20250624 14:19:07.380841 4261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb [sys.catalog]: This master's current role is: LEADER
I20250624 14:19:07.382942 4269 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 14:19:07.395202 4269 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 14:19:07.411223 4269 catalog_manager.cc:1349] Generated new cluster ID: b4630439387646a588f710ab08aa07d6
I20250624 14:19:07.411582 4269 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 14:19:07.436776 4269 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 14:19:07.438344 4269 catalog_manager.cc:1506] Loading token signing keys...
I20250624 14:19:07.458861 4269 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 27ff67805a64429db27c904ac59feecb: Generated new TSK 0
I20250624 14:19:07.459957 4269 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 14:19:07.473345 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.1:0
--local_ip_for_outbound_sockets=127.2.14.1
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41831
--builtin_ntp_servers=127.2.14.20:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 14:19:07.796113 4278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:07.796643 4278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:07.797179 4278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:07.829310 4278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:07.830279 4278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.1
I20250624 14:19:07.866240 4278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.1
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:07.867688 4278 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:07.869392 4278 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:07.888142 4285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:07.891323 4287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:07.889592 4284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:09.382719 4286 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1492 milliseconds
I20250624 14:19:09.382886 4278 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:19:09.384618 4278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:09.387691 4278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:09.389192 4278 hybrid_clock.cc:648] HybridClock initialized: now 1750774749389137 us; error 56 us; skew 500 ppm
I20250624 14:19:09.390326 4278 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:09.403249 4278 webserver.cc:469] Webserver started at http://127.2.14.1:33085/ using document root <none> and password file <none>
I20250624 14:19:09.404610 4278 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:09.404906 4278 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:09.405522 4278 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:19:09.412308 4278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e43653d73fa4483eba6b1de34715bf77"
format_stamp: "Formatted at 2025-06-24 14:19:09 on dist-test-slave-hs17"
I20250624 14:19:09.413903 4278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e43653d73fa4483eba6b1de34715bf77"
format_stamp: "Formatted at 2025-06-24 14:19:09 on dist-test-slave-hs17"
I20250624 14:19:09.423467 4278 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.003s
I20250624 14:19:09.431071 4294 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:09.432360 4278 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250624 14:19:09.432750 4278 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e43653d73fa4483eba6b1de34715bf77"
format_stamp: "Formatted at 2025-06-24 14:19:09 on dist-test-slave-hs17"
I20250624 14:19:09.433207 4278 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:09.497327 4278 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:09.499359 4278 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:09.499997 4278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:09.503333 4278 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:19:09.509117 4278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:19:09.509419 4278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:09.509732 4278 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:19:09.509953 4278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:09.667644 4278 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.1:44059
I20250624 14:19:09.667742 4406 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.1:44059 every 8 connection(s)
I20250624 14:19:09.671221 4278 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 14:19:09.676801 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 4278
I20250624 14:19:09.677307 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 14:19:09.687876 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.2:0
--local_ip_for_outbound_sockets=127.2.14.2
--webserver_interface=127.2.14.2
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41831
--builtin_ntp_servers=127.2.14.20:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:19:09.700521 4407 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41831
I20250624 14:19:09.700958 4407 heartbeater.cc:461] Registering TS with master...
I20250624 14:19:09.701992 4407 heartbeater.cc:507] Master 127.2.14.62:41831 requested a full tablet report, sending...
I20250624 14:19:09.704452 4219 ts_manager.cc:194] Registered new tserver with Master: e43653d73fa4483eba6b1de34715bf77 (127.2.14.1:44059)
I20250624 14:19:09.706450 4219 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.1:44899
W20250624 14:19:10.032797 4411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:10.033421 4411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:10.034173 4411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:10.073029 4411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:10.073923 4411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.2
I20250624 14:19:10.110567 4411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.14.2
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.2
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:10.112033 4411 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:10.113706 4411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:10.129422 4417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:10.709895 4407 heartbeater.cc:499] Master 127.2.14.62:41831 was elected leader, sending a full tablet report...
W20250624 14:19:10.129621 4418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:10.132081 4411 server_base.cc:1048] running on GCE node
W20250624 14:19:10.130638 4420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:11.313506 4411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:11.315872 4411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:11.317250 4411 hybrid_clock.cc:648] HybridClock initialized: now 1750774751317184 us; error 71 us; skew 500 ppm
I20250624 14:19:11.318061 4411 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:11.324520 4411 webserver.cc:469] Webserver started at http://127.2.14.2:38333/ using document root <none> and password file <none>
I20250624 14:19:11.325486 4411 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:11.325711 4411 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:11.326185 4411 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:19:11.331388 4411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "73909fcef6b94b60b53b6cd3ea640c1e"
format_stamp: "Formatted at 2025-06-24 14:19:11 on dist-test-slave-hs17"
I20250624 14:19:11.332602 4411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "73909fcef6b94b60b53b6cd3ea640c1e"
format_stamp: "Formatted at 2025-06-24 14:19:11 on dist-test-slave-hs17"
I20250624 14:19:11.340142 4411 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250624 14:19:11.345827 4427 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:11.346920 4411 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250624 14:19:11.347246 4411 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "73909fcef6b94b60b53b6cd3ea640c1e"
format_stamp: "Formatted at 2025-06-24 14:19:11 on dist-test-slave-hs17"
I20250624 14:19:11.347626 4411 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:11.403681 4411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:11.405229 4411 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:11.405675 4411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:11.408313 4411 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:19:11.412786 4411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:19:11.413008 4411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:11.413264 4411 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:19:11.413426 4411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:11.551210 4411 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.2:34389
I20250624 14:19:11.551319 4539 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.2:34389 every 8 connection(s)
I20250624 14:19:11.553900 4411 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 14:19:11.562520 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 4411
I20250624 14:19:11.562915 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 14:19:11.568884 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.3:0
--local_ip_for_outbound_sockets=127.2.14.3
--webserver_interface=127.2.14.3
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41831
--builtin_ntp_servers=127.2.14.20:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:19:11.575765 4540 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41831
I20250624 14:19:11.576438 4540 heartbeater.cc:461] Registering TS with master...
I20250624 14:19:11.577690 4540 heartbeater.cc:507] Master 127.2.14.62:41831 requested a full tablet report, sending...
I20250624 14:19:11.580457 4219 ts_manager.cc:194] Registered new tserver with Master: 73909fcef6b94b60b53b6cd3ea640c1e (127.2.14.2:34389)
I20250624 14:19:11.582311 4219 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.2:37251
W20250624 14:19:11.887805 4544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:11.888331 4544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:11.888833 4544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:11.921907 4544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:11.922778 4544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.3
I20250624 14:19:11.958588 4544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.14.3
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.3
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:11.960060 4544 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:11.961802 4544 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:11.977833 4551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:12.586304 4540 heartbeater.cc:499] Master 127.2.14.62:41831 was elected leader, sending a full tablet report...
W20250624 14:19:11.978049 4552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:11.979974 4554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:11.980306 4544 server_base.cc:1048] running on GCE node
I20250624 14:19:13.152570 4544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:13.155483 4544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:13.156997 4544 hybrid_clock.cc:648] HybridClock initialized: now 1750774753156944 us; error 69 us; skew 500 ppm
I20250624 14:19:13.157826 4544 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:13.164664 4544 webserver.cc:469] Webserver started at http://127.2.14.3:42661/ using document root <none> and password file <none>
I20250624 14:19:13.165644 4544 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:13.165858 4544 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:13.166420 4544 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:19:13.170948 4544 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4ae8d70d757a41e7adc8a236cfb981c2"
format_stamp: "Formatted at 2025-06-24 14:19:13 on dist-test-slave-hs17"
I20250624 14:19:13.172075 4544 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4ae8d70d757a41e7adc8a236cfb981c2"
format_stamp: "Formatted at 2025-06-24 14:19:13 on dist-test-slave-hs17"
I20250624 14:19:13.179375 4544 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20250624 14:19:13.185294 4561 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:13.186368 4544 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250624 14:19:13.186683 4544 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4ae8d70d757a41e7adc8a236cfb981c2"
format_stamp: "Formatted at 2025-06-24 14:19:13 on dist-test-slave-hs17"
I20250624 14:19:13.187048 4544 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:13.237124 4544 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:13.238574 4544 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:13.239017 4544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:13.241505 4544 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:19:13.245543 4544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:19:13.245750 4544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:13.246022 4544 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:19:13.246169 4544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:13.384410 4544 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.3:44407
I20250624 14:19:13.384524 4673 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.3:44407 every 8 connection(s)
I20250624 14:19:13.387077 4544 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 14:19:13.388942 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 4544
I20250624 14:19:13.389532 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 14:19:13.409819 4674 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41831
I20250624 14:19:13.410367 4674 heartbeater.cc:461] Registering TS with master...
I20250624 14:19:13.411579 4674 heartbeater.cc:507] Master 127.2.14.62:41831 requested a full tablet report, sending...
I20250624 14:19:13.413620 4218 ts_manager.cc:194] Registered new tserver with Master: 4ae8d70d757a41e7adc8a236cfb981c2 (127.2.14.3:44407)
I20250624 14:19:13.414988 4218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.3:46291
I20250624 14:19:13.425783 2104 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 14:19:13.460557 4218 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47998:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250624 14:19:13.479794 4218 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 14:19:13.539485 4342 tablet_service.cc:1468] Processing CreateTablet for tablet bff7d8ed902b463f80daf8d1942cbae9 (DEFAULT_TABLE table=TestTable [id=26ca0775ca7f4e1692a70d2750c94f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:19:13.542572 4342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bff7d8ed902b463f80daf8d1942cbae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:19:13.542589 4609 tablet_service.cc:1468] Processing CreateTablet for tablet bff7d8ed902b463f80daf8d1942cbae9 (DEFAULT_TABLE table=TestTable [id=26ca0775ca7f4e1692a70d2750c94f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:19:13.542800 4475 tablet_service.cc:1468] Processing CreateTablet for tablet bff7d8ed902b463f80daf8d1942cbae9 (DEFAULT_TABLE table=TestTable [id=26ca0775ca7f4e1692a70d2750c94f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:19:13.544428 4609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bff7d8ed902b463f80daf8d1942cbae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:19:13.544574 4475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bff7d8ed902b463f80daf8d1942cbae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:19:13.581053 4693 tablet_bootstrap.cc:492] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77: Bootstrap starting.
I20250624 14:19:13.581970 4694 tablet_bootstrap.cc:492] T bff7d8ed902b463f80daf8d1942cbae9 P 4ae8d70d757a41e7adc8a236cfb981c2: Bootstrap starting.
I20250624 14:19:13.587853 4695 tablet_bootstrap.cc:492] T bff7d8ed902b463f80daf8d1942cbae9 P 73909fcef6b94b60b53b6cd3ea640c1e: Bootstrap starting.
I20250624 14:19:13.588078 4693 tablet_bootstrap.cc:654] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77: Neither blocks nor log segments found. Creating new log.
I20250624 14:19:13.590569 4693 log.cc:826] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:13.595606 4695 tablet_bootstrap.cc:654] T bff7d8ed902b463f80daf8d1942cbae9 P 73909fcef6b94b60b53b6cd3ea640c1e: Neither blocks nor log segments found. Creating new log.
I20250624 14:19:13.598882 4694 tablet_bootstrap.cc:654] T bff7d8ed902b463f80daf8d1942cbae9 P 4ae8d70d757a41e7adc8a236cfb981c2: Neither blocks nor log segments found. Creating new log.
I20250624 14:19:13.599071 4695 log.cc:826] T bff7d8ed902b463f80daf8d1942cbae9 P 73909fcef6b94b60b53b6cd3ea640c1e: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:13.599228 4693 tablet_bootstrap.cc:492] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77: No bootstrap required, opened a new log
I20250624 14:19:13.599854 4693 ts_tablet_manager.cc:1397] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.006s
I20250624 14:19:13.601596 4694 log.cc:826] T bff7d8ed902b463f80daf8d1942cbae9 P 4ae8d70d757a41e7adc8a236cfb981c2: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:13.607403 4695 tablet_bootstrap.cc:492] T bff7d8ed902b463f80daf8d1942cbae9 P 73909fcef6b94b60b53b6cd3ea640c1e: No bootstrap required, opened a new log
I20250624 14:19:13.608083 4695 ts_tablet_manager.cc:1397] T bff7d8ed902b463f80daf8d1942cbae9 P 73909fcef6b94b60b53b6cd3ea640c1e: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.003s
I20250624 14:19:13.611047 4694 tablet_bootstrap.cc:492] T bff7d8ed902b463f80daf8d1942cbae9 P 4ae8d70d757a41e7adc8a236cfb981c2: No bootstrap required, opened a new log
I20250624 14:19:13.611604 4694 ts_tablet_manager.cc:1397] T bff7d8ed902b463f80daf8d1942cbae9 P 4ae8d70d757a41e7adc8a236cfb981c2: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.006s
I20250624 14:19:13.627225 4695 raft_consensus.cc:357] T bff7d8ed902b463f80daf8d1942cbae9 P 73909fcef6b94b60b53b6cd3ea640c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73909fcef6b94b60b53b6cd3ea640c1e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 34389 } } peers { permanent_uuid: "e43653d73fa4483eba6b1de34715bf77" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 44059 } } peers { permanent_uuid: "4ae8d70d757a41e7adc8a236cfb981c2" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 44407 } }
I20250624 14:19:13.628041 4695 raft_consensus.cc:383] T bff7d8ed902b463f80daf8d1942cbae9 P 73909fcef6b94b60b53b6cd3ea640c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:19:13.628325 4695 raft_consensus.cc:738] T bff7d8ed902b463f80daf8d1942cbae9 P 73909fcef6b94b60b53b6cd3ea640c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73909fcef6b94b60b53b6cd3ea640c1e, State: Initialized, Role: FOLLOWER
I20250624 14:19:13.628588 4693 raft_consensus.cc:357] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73909fcef6b94b60b53b6cd3ea640c1e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 34389 } } peers { permanent_uuid: "e43653d73fa4483eba6b1de34715bf77" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 44059 } } peers { permanent_uuid: "4ae8d70d757a41e7adc8a236cfb981c2" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 44407 } }
I20250624 14:19:13.629112 4695 consensus_queue.cc:260] T bff7d8ed902b463f80daf8d1942cbae9 P 73909fcef6b94b60b53b6cd3ea640c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73909fcef6b94b60b53b6cd3ea640c1e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 34389 } } peers { permanent_uuid: "e43653d73fa4483eba6b1de34715bf77" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 44059 } } peers { permanent_uuid: "4ae8d70d757a41e7adc8a236cfb981c2" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 44407 } }
I20250624 14:19:13.629485 4693 raft_consensus.cc:383] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:19:13.629835 4693 raft_consensus.cc:738] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e43653d73fa4483eba6b1de34715bf77, State: Initialized, Role: FOLLOWER
I20250624 14:19:13.630779 4693 consensus_queue.cc:260] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73909fcef6b94b60b53b6cd3ea640c1e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 34389 } } peers { permanent_uuid: "e43653d73fa4483eba6b1de34715bf77" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 44059 } } peers { permanent_uuid: "4ae8d70d757a41e7adc8a236cfb981c2" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 44407 } }
I20250624 14:19:13.633584 4695 ts_tablet_manager.cc:1428] T bff7d8ed902b463f80daf8d1942cbae9 P 73909fcef6b94b60b53b6cd3ea640c1e: Time spent starting tablet: real 0.025s user 0.020s sys 0.004s
I20250624 14:19:13.640985 4693 ts_tablet_manager.cc:1428] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77: Time spent starting tablet: real 0.041s user 0.022s sys 0.016s
I20250624 14:19:13.640952 4694 raft_consensus.cc:357] T bff7d8ed902b463f80daf8d1942cbae9 P 4ae8d70d757a41e7adc8a236cfb981c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73909fcef6b94b60b53b6cd3ea640c1e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 34389 } } peers { permanent_uuid: "e43653d73fa4483eba6b1de34715bf77" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 44059 } } peers { permanent_uuid: "4ae8d70d757a41e7adc8a236cfb981c2" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 44407 } }
I20250624 14:19:13.641881 4694 raft_consensus.cc:383] T bff7d8ed902b463f80daf8d1942cbae9 P 4ae8d70d757a41e7adc8a236cfb981c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:19:13.642273 4694 raft_consensus.cc:738] T bff7d8ed902b463f80daf8d1942cbae9 P 4ae8d70d757a41e7adc8a236cfb981c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae8d70d757a41e7adc8a236cfb981c2, State: Initialized, Role: FOLLOWER
I20250624 14:19:13.643229 4694 consensus_queue.cc:260] T bff7d8ed902b463f80daf8d1942cbae9 P 4ae8d70d757a41e7adc8a236cfb981c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73909fcef6b94b60b53b6cd3ea640c1e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 34389 } } peers { permanent_uuid: "e43653d73fa4483eba6b1de34715bf77" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 44059 } } peers { permanent_uuid: "4ae8d70d757a41e7adc8a236cfb981c2" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 44407 } }
I20250624 14:19:13.647399 4674 heartbeater.cc:499] Master 127.2.14.62:41831 was elected leader, sending a full tablet report...
I20250624 14:19:13.647747 4694 ts_tablet_manager.cc:1428] T bff7d8ed902b463f80daf8d1942cbae9 P 4ae8d70d757a41e7adc8a236cfb981c2: Time spent starting tablet: real 0.036s user 0.031s sys 0.005s
I20250624 14:19:13.673204 4700 raft_consensus.cc:491] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 14:19:13.673722 4700 raft_consensus.cc:513] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73909fcef6b94b60b53b6cd3ea640c1e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 34389 } } peers { permanent_uuid: "e43653d73fa4483eba6b1de34715bf77" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 44059 } } peers { permanent_uuid: "4ae8d70d757a41e7adc8a236cfb981c2" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 44407 } }
I20250624 14:19:13.676409 4700 leader_election.cc:290] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73909fcef6b94b60b53b6cd3ea640c1e (127.2.14.2:34389), 4ae8d70d757a41e7adc8a236cfb981c2 (127.2.14.3:44407)
W20250624 14:19:13.681128 4408 tablet.cc:2378] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:19:13.692129 4495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bff7d8ed902b463f80daf8d1942cbae9" candidate_uuid: "e43653d73fa4483eba6b1de34715bf77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73909fcef6b94b60b53b6cd3ea640c1e" is_pre_election: true
I20250624 14:19:13.693097 4495 raft_consensus.cc:2466] T bff7d8ed902b463f80daf8d1942cbae9 P 73909fcef6b94b60b53b6cd3ea640c1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e43653d73fa4483eba6b1de34715bf77 in term 0.
I20250624 14:19:13.694648 4296 leader_election.cc:304] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73909fcef6b94b60b53b6cd3ea640c1e, e43653d73fa4483eba6b1de34715bf77; no voters:
I20250624 14:19:13.694973 4629 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bff7d8ed902b463f80daf8d1942cbae9" candidate_uuid: "e43653d73fa4483eba6b1de34715bf77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae8d70d757a41e7adc8a236cfb981c2" is_pre_election: true
I20250624 14:19:13.695605 4700 raft_consensus.cc:2802] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 14:19:13.695971 4700 raft_consensus.cc:491] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 14:19:13.695883 4629 raft_consensus.cc:2466] T bff7d8ed902b463f80daf8d1942cbae9 P 4ae8d70d757a41e7adc8a236cfb981c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e43653d73fa4483eba6b1de34715bf77 in term 0.
I20250624 14:19:13.696298 4700 raft_consensus.cc:3058] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:19:13.701015 4700 raft_consensus.cc:513] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73909fcef6b94b60b53b6cd3ea640c1e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 34389 } } peers { permanent_uuid: "e43653d73fa4483eba6b1de34715bf77" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 44059 } } peers { permanent_uuid: "4ae8d70d757a41e7adc8a236cfb981c2" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 44407 } }
I20250624 14:19:13.702737 4700 leader_election.cc:290] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [CANDIDATE]: Term 1 election: Requested vote from peers 73909fcef6b94b60b53b6cd3ea640c1e (127.2.14.2:34389), 4ae8d70d757a41e7adc8a236cfb981c2 (127.2.14.3:44407)
I20250624 14:19:13.703604 4495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bff7d8ed902b463f80daf8d1942cbae9" candidate_uuid: "e43653d73fa4483eba6b1de34715bf77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73909fcef6b94b60b53b6cd3ea640c1e"
I20250624 14:19:13.703897 4629 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bff7d8ed902b463f80daf8d1942cbae9" candidate_uuid: "e43653d73fa4483eba6b1de34715bf77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae8d70d757a41e7adc8a236cfb981c2"
I20250624 14:19:13.704181 4495 raft_consensus.cc:3058] T bff7d8ed902b463f80daf8d1942cbae9 P 73909fcef6b94b60b53b6cd3ea640c1e [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:19:13.704444 4629 raft_consensus.cc:3058] T bff7d8ed902b463f80daf8d1942cbae9 P 4ae8d70d757a41e7adc8a236cfb981c2 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:19:13.711433 4495 raft_consensus.cc:2466] T bff7d8ed902b463f80daf8d1942cbae9 P 73909fcef6b94b60b53b6cd3ea640c1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e43653d73fa4483eba6b1de34715bf77 in term 1.
I20250624 14:19:13.711603 4629 raft_consensus.cc:2466] T bff7d8ed902b463f80daf8d1942cbae9 P 4ae8d70d757a41e7adc8a236cfb981c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e43653d73fa4483eba6b1de34715bf77 in term 1.
I20250624 14:19:13.712837 4297 leader_election.cc:304] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae8d70d757a41e7adc8a236cfb981c2, e43653d73fa4483eba6b1de34715bf77; no voters:
I20250624 14:19:13.713585 4700 raft_consensus.cc:2802] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:19:13.715134 4700 raft_consensus.cc:695] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [term 1 LEADER]: Becoming Leader. State: Replica: e43653d73fa4483eba6b1de34715bf77, State: Running, Role: LEADER
I20250624 14:19:13.715905 4700 consensus_queue.cc:237] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73909fcef6b94b60b53b6cd3ea640c1e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 34389 } } peers { permanent_uuid: "e43653d73fa4483eba6b1de34715bf77" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 44059 } } peers { permanent_uuid: "4ae8d70d757a41e7adc8a236cfb981c2" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 44407 } }
I20250624 14:19:13.727821 4218 catalog_manager.cc:5582] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 reported cstate change: term changed from 0 to 1, leader changed from <none> to e43653d73fa4483eba6b1de34715bf77 (127.2.14.1). New cstate: current_term: 1 leader_uuid: "e43653d73fa4483eba6b1de34715bf77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73909fcef6b94b60b53b6cd3ea640c1e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 34389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e43653d73fa4483eba6b1de34715bf77" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 44059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ae8d70d757a41e7adc8a236cfb981c2" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 44407 } health_report { overall_health: UNKNOWN } } }
I20250624 14:19:13.749879 2104 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 14:19:13.753295 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e43653d73fa4483eba6b1de34715bf77 to finish bootstrapping
I20250624 14:19:13.765676 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 73909fcef6b94b60b53b6cd3ea640c1e to finish bootstrapping
I20250624 14:19:13.776525 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4ae8d70d757a41e7adc8a236cfb981c2 to finish bootstrapping
W20250624 14:19:13.810750 4541 tablet.cc:2378] T bff7d8ed902b463f80daf8d1942cbae9 P 73909fcef6b94b60b53b6cd3ea640c1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 14:19:13.892581 4675 tablet.cc:2378] T bff7d8ed902b463f80daf8d1942cbae9 P 4ae8d70d757a41e7adc8a236cfb981c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:19:14.224574 4700 consensus_queue.cc:1035] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73909fcef6b94b60b53b6cd3ea640c1e" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 34389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250624 14:19:14.259143 4719 consensus_queue.cc:1035] T bff7d8ed902b463f80daf8d1942cbae9 P e43653d73fa4483eba6b1de34715bf77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ae8d70d757a41e7adc8a236cfb981c2" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250624 14:19:15.578526 4218 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:48014
I20250624 14:19:16.609826 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 4278
I20250624 14:19:16.637472 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 4411
I20250624 14:19:16.663435 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 4544
I20250624 14:19:16.692333 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 4186
2025-06-24T14:19:16Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (11260 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250624 14:19:16.746181 2104 test_util.cc:276] Using random seed: 58008482
I20250624 14:19:16.750361 2104 ts_itest-base.cc:115] Starting cluster with:
I20250624 14:19:16.750568 2104 ts_itest-base.cc:116] --------------
I20250624 14:19:16.750744 2104 ts_itest-base.cc:117] 3 tablet servers
I20250624 14:19:16.750885 2104 ts_itest-base.cc:118] 3 replicas per TS
I20250624 14:19:16.751021 2104 ts_itest-base.cc:119] --------------
2025-06-24T14:19:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T14:19:16Z Disabled control of system clock
I20250624 14:19:16.791960 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:41603
--webserver_interface=127.2.14.62
--webserver_port=0
--builtin_ntp_servers=127.2.14.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.14.62:41603 with env {}
W20250624 14:19:17.092262 4745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:17.092820 4745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:17.093231 4745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:17.124801 4745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 14:19:17.125102 4745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:17.125304 4745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 14:19:17.125499 4745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 14:19:17.160851 4745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.14.62:41603
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:41603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:17.162132 4745 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:17.163822 4745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:17.177639 4751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:17.179698 4754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:17.177754 4752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:18.372646 4753 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1192 milliseconds
I20250624 14:19:18.372793 4745 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:19:18.374063 4745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:18.377210 4745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:18.378692 4745 hybrid_clock.cc:648] HybridClock initialized: now 1750774758378647 us; error 56 us; skew 500 ppm
I20250624 14:19:18.379514 4745 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:18.386703 4745 webserver.cc:469] Webserver started at http://127.2.14.62:34921/ using document root <none> and password file <none>
I20250624 14:19:18.387723 4745 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:18.387930 4745 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:18.388370 4745 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:19:18.392928 4745 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7db2648ab6e441c38549ea70be0355e3"
format_stamp: "Formatted at 2025-06-24 14:19:18 on dist-test-slave-hs17"
I20250624 14:19:18.394034 4745 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7db2648ab6e441c38549ea70be0355e3"
format_stamp: "Formatted at 2025-06-24 14:19:18 on dist-test-slave-hs17"
I20250624 14:19:18.401530 4745 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250624 14:19:18.407439 4761 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:18.408565 4745 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250624 14:19:18.408887 4745 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7db2648ab6e441c38549ea70be0355e3"
format_stamp: "Formatted at 2025-06-24 14:19:18 on dist-test-slave-hs17"
I20250624 14:19:18.409215 4745 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:18.463496 4745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:18.465014 4745 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:18.465417 4745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:18.536517 4745 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.62:41603
I20250624 14:19:18.536651 4812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.62:41603 every 8 connection(s)
I20250624 14:19:18.539424 4745 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 14:19:18.544401 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 4745
I20250624 14:19:18.544940 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 14:19:18.545399 4813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:19:18.571097 4813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Bootstrap starting.
I20250624 14:19:18.576848 4813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Neither blocks nor log segments found. Creating new log.
I20250624 14:19:18.578619 4813 log.cc:826] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:18.583201 4813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: No bootstrap required, opened a new log
I20250624 14:19:18.601238 4813 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:18.601917 4813 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:19:18.602203 4813 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db2648ab6e441c38549ea70be0355e3, State: Initialized, Role: FOLLOWER
I20250624 14:19:18.602939 4813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:18.603471 4813 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:19:18.603808 4813 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:19:18.604112 4813 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:19:18.609686 4813 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:18.610528 4813 leader_election.cc:304] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7db2648ab6e441c38549ea70be0355e3; no voters:
I20250624 14:19:18.612206 4813 leader_election.cc:290] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 14:19:18.612928 4818 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:19:18.615139 4818 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 1 LEADER]: Becoming Leader. State: Replica: 7db2648ab6e441c38549ea70be0355e3, State: Running, Role: LEADER
I20250624 14:19:18.615959 4818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:18.617105 4813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 14:19:18.628050 4820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7db2648ab6e441c38549ea70be0355e3. Latest consensus state: current_term: 1 leader_uuid: "7db2648ab6e441c38549ea70be0355e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } } }
I20250624 14:19:18.628295 4819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7db2648ab6e441c38549ea70be0355e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } } }
I20250624 14:19:18.628926 4820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: This master's current role is: LEADER
I20250624 14:19:18.629325 4819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: This master's current role is: LEADER
I20250624 14:19:18.632516 4827 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 14:19:18.644091 4827 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 14:19:18.662114 4827 catalog_manager.cc:1349] Generated new cluster ID: db11613b64224d6a861fb54fb06c5787
I20250624 14:19:18.662447 4827 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 14:19:18.707525 4827 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 14:19:18.709668 4827 catalog_manager.cc:1506] Loading token signing keys...
I20250624 14:19:18.725644 4827 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Generated new TSK 0
I20250624 14:19:18.726792 4827 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 14:19:18.746085 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.1:0
--local_ip_for_outbound_sockets=127.2.14.1
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41603
--builtin_ntp_servers=127.2.14.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 14:19:19.057940 4837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:19.058472 4837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:19.058990 4837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:19.091363 4837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:19.092262 4837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.1
I20250624 14:19:19.128301 4837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41603
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.1
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:19.129760 4837 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:19.131609 4837 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:19.149477 4844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:19.152336 4843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:19.149529 4846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:20.602943 4845 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1450 milliseconds
I20250624 14:19:20.603042 4837 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:19:20.604450 4837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:20.608247 4837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:20.609704 4837 hybrid_clock.cc:648] HybridClock initialized: now 1750774760609642 us; error 81 us; skew 500 ppm
I20250624 14:19:20.610513 4837 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:20.621515 4837 webserver.cc:469] Webserver started at http://127.2.14.1:41237/ using document root <none> and password file <none>
I20250624 14:19:20.622848 4837 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:20.623148 4837 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:20.623790 4837 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:19:20.630584 4837 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "55a21f9c8e154fcc85e4b33617f4aec7"
format_stamp: "Formatted at 2025-06-24 14:19:20 on dist-test-slave-hs17"
I20250624 14:19:20.632192 4837 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "55a21f9c8e154fcc85e4b33617f4aec7"
format_stamp: "Formatted at 2025-06-24 14:19:20 on dist-test-slave-hs17"
I20250624 14:19:20.641777 4837 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20250624 14:19:20.649587 4853 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:20.650873 4837 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250624 14:19:20.651314 4837 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "55a21f9c8e154fcc85e4b33617f4aec7"
format_stamp: "Formatted at 2025-06-24 14:19:20 on dist-test-slave-hs17"
I20250624 14:19:20.651813 4837 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:20.719290 4837 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:20.720924 4837 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:20.721377 4837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:20.724566 4837 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:19:20.729192 4837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:19:20.729427 4837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:20.729687 4837 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:19:20.729862 4837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:20.902107 4837 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.1:36801
I20250624 14:19:20.902287 4965 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.1:36801 every 8 connection(s)
I20250624 14:19:20.904723 4837 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 14:19:20.911482 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 4837
I20250624 14:19:20.912024 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 14:19:20.919750 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.2:0
--local_ip_for_outbound_sockets=127.2.14.2
--webserver_interface=127.2.14.2
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41603
--builtin_ntp_servers=127.2.14.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:19:20.931720 4966 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41603
I20250624 14:19:20.932267 4966 heartbeater.cc:461] Registering TS with master...
I20250624 14:19:20.933630 4966 heartbeater.cc:507] Master 127.2.14.62:41603 requested a full tablet report, sending...
I20250624 14:19:20.936830 4778 ts_manager.cc:194] Registered new tserver with Master: 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1:36801)
I20250624 14:19:20.939671 4778 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.1:58893
W20250624 14:19:21.236989 4970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:21.237527 4970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:21.238044 4970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:21.270306 4970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:21.271214 4970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.2
I20250624 14:19:21.306432 4970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.14.2
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41603
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.2
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:21.307852 4970 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:21.309540 4970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:21.326138 4977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:21.944116 4966 heartbeater.cc:499] Master 127.2.14.62:41603 was elected leader, sending a full tablet report...
W20250624 14:19:21.326577 4976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:21.326648 4979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:22.493649 4970 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250624 14:19:22.493683 4978 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 14:19:22.498240 4970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:22.500520 4970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:22.501924 4970 hybrid_clock.cc:648] HybridClock initialized: now 1750774762501885 us; error 48 us; skew 500 ppm
I20250624 14:19:22.502763 4970 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:22.509730 4970 webserver.cc:469] Webserver started at http://127.2.14.2:45297/ using document root <none> and password file <none>
I20250624 14:19:22.510671 4970 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:22.510915 4970 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:22.511368 4970 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:19:22.515914 4970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4112eeddb57742e48ba4b705d3a0c0a7"
format_stamp: "Formatted at 2025-06-24 14:19:22 on dist-test-slave-hs17"
I20250624 14:19:22.517017 4970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4112eeddb57742e48ba4b705d3a0c0a7"
format_stamp: "Formatted at 2025-06-24 14:19:22 on dist-test-slave-hs17"
I20250624 14:19:22.524459 4970 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250624 14:19:22.530099 4987 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:22.531121 4970 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250624 14:19:22.531453 4970 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4112eeddb57742e48ba4b705d3a0c0a7"
format_stamp: "Formatted at 2025-06-24 14:19:22 on dist-test-slave-hs17"
I20250624 14:19:22.531842 4970 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:22.581423 4970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:22.582935 4970 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:22.583370 4970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:22.585896 4970 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:19:22.590160 4970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:19:22.590384 4970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:22.590647 4970 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:19:22.590821 4970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:22.729516 4970 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.2:39265
I20250624 14:19:22.729650 5099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.2:39265 every 8 connection(s)
I20250624 14:19:22.732143 4970 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 14:19:22.742340 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 4970
I20250624 14:19:22.742959 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 14:19:22.750658 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.3:0
--local_ip_for_outbound_sockets=127.2.14.3
--webserver_interface=127.2.14.3
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41603
--builtin_ntp_servers=127.2.14.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:19:22.754766 5100 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41603
I20250624 14:19:22.755209 5100 heartbeater.cc:461] Registering TS with master...
I20250624 14:19:22.756284 5100 heartbeater.cc:507] Master 127.2.14.62:41603 requested a full tablet report, sending...
I20250624 14:19:22.758414 4778 ts_manager.cc:194] Registered new tserver with Master: 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2:39265)
I20250624 14:19:22.759974 4778 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.2:50235
W20250624 14:19:23.058054 5104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:23.058571 5104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:23.059067 5104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:23.090397 5104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:23.091272 5104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.3
I20250624 14:19:23.126278 5104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.14.3
--webserver_port=0
--tserver_master_addrs=127.2.14.62:41603
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.3
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:23.127624 5104 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:23.129359 5104 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:23.146003 5110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:23.764120 5100 heartbeater.cc:499] Master 127.2.14.62:41603 was elected leader, sending a full tablet report...
W20250624 14:19:23.146423 5111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:23.146725 5113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:23.148164 5104 server_base.cc:1048] running on GCE node
I20250624 14:19:24.318586 5104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:24.321007 5104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:24.322393 5104 hybrid_clock.cc:648] HybridClock initialized: now 1750774764322342 us; error 59 us; skew 500 ppm
I20250624 14:19:24.323232 5104 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:24.329632 5104 webserver.cc:469] Webserver started at http://127.2.14.3:34735/ using document root <none> and password file <none>
I20250624 14:19:24.330587 5104 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:24.330828 5104 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:24.331281 5104 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:19:24.336037 5104 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "60919fec2a7d41739f65174c4e92014c"
format_stamp: "Formatted at 2025-06-24 14:19:24 on dist-test-slave-hs17"
I20250624 14:19:24.337167 5104 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "60919fec2a7d41739f65174c4e92014c"
format_stamp: "Formatted at 2025-06-24 14:19:24 on dist-test-slave-hs17"
I20250624 14:19:24.345237 5104 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250624 14:19:24.350932 5120 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:24.352380 5104 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250624 14:19:24.352710 5104 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "60919fec2a7d41739f65174c4e92014c"
format_stamp: "Formatted at 2025-06-24 14:19:24 on dist-test-slave-hs17"
I20250624 14:19:24.353060 5104 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:24.410544 5104 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:24.412269 5104 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:24.412778 5104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:24.415275 5104 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:19:24.420107 5104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:19:24.420428 5104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:24.420697 5104 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:19:24.420850 5104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:24.562337 5104 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.3:42843
I20250624 14:19:24.562430 5232 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.3:42843 every 8 connection(s)
I20250624 14:19:24.565796 5104 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 14:19:24.569609 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 5104
I20250624 14:19:24.570119 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 14:19:24.594264 5233 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41603
I20250624 14:19:24.594669 5233 heartbeater.cc:461] Registering TS with master...
I20250624 14:19:24.595732 5233 heartbeater.cc:507] Master 127.2.14.62:41603 requested a full tablet report, sending...
I20250624 14:19:24.597851 4778 ts_manager.cc:194] Registered new tserver with Master: 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843)
I20250624 14:19:24.599120 4778 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.3:51675
I20250624 14:19:24.607018 2104 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 14:19:24.644060 2104 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 14:19:24.644371 2104 test_util.cc:276] Using random seed: 65906689
I20250624 14:19:24.686100 4778 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37962:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 14:19:24.729889 4901 tablet_service.cc:1468] Processing CreateTablet for tablet df16f2e37e1e471db624a209f2ade936 (DEFAULT_TABLE table=TestTable [id=e8f33b3ba75144dca510cc4ca0a50374]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:19:24.732028 4901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df16f2e37e1e471db624a209f2ade936. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:19:24.754324 5253 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Bootstrap starting.
I20250624 14:19:24.760071 5253 tablet_bootstrap.cc:654] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Neither blocks nor log segments found. Creating new log.
I20250624 14:19:24.761895 5253 log.cc:826] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:24.766314 5253 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: No bootstrap required, opened a new log
I20250624 14:19:24.766750 5253 ts_tablet_manager.cc:1397] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20250624 14:19:24.784404 5253 raft_consensus.cc:357] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } }
I20250624 14:19:24.784993 5253 raft_consensus.cc:383] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:19:24.785283 5253 raft_consensus.cc:738] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a21f9c8e154fcc85e4b33617f4aec7, State: Initialized, Role: FOLLOWER
I20250624 14:19:24.786120 5253 consensus_queue.cc:260] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } }
I20250624 14:19:24.786636 5253 raft_consensus.cc:397] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:19:24.786887 5253 raft_consensus.cc:491] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:19:24.787147 5253 raft_consensus.cc:3058] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:19:24.791281 5253 raft_consensus.cc:513] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } }
I20250624 14:19:24.792027 5253 leader_election.cc:304] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55a21f9c8e154fcc85e4b33617f4aec7; no voters:
I20250624 14:19:24.793706 5253 leader_election.cc:290] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 14:19:24.794103 5255 raft_consensus.cc:2802] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:19:24.796677 5255 raft_consensus.cc:695] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 1 LEADER]: Becoming Leader. State: Replica: 55a21f9c8e154fcc85e4b33617f4aec7, State: Running, Role: LEADER
I20250624 14:19:24.797606 5255 consensus_queue.cc:237] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } }
I20250624 14:19:24.798390 5253 ts_tablet_manager.cc:1428] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Time spent starting tablet: real 0.031s user 0.025s sys 0.007s
I20250624 14:19:24.809937 4778 catalog_manager.cc:5582] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1). New cstate: current_term: 1 leader_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } health_report { overall_health: HEALTHY } } }
I20250624 14:19:25.020020 2104 test_util.cc:276] Using random seed: 66282326
I20250624 14:19:25.042275 4778 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37972:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 14:19:25.071120 5035 tablet_service.cc:1468] Processing CreateTablet for tablet 678437be284e4cf29a9dbb50aaa3727d (DEFAULT_TABLE table=TestTable1 [id=279a7e12430d4deea6cef76d98cc4de6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:19:25.072633 5035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678437be284e4cf29a9dbb50aaa3727d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:19:25.095896 5274 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Bootstrap starting.
I20250624 14:19:25.101545 5274 tablet_bootstrap.cc:654] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Neither blocks nor log segments found. Creating new log.
I20250624 14:19:25.103271 5274 log.cc:826] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:25.108291 5274 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: No bootstrap required, opened a new log
I20250624 14:19:25.108701 5274 ts_tablet_manager.cc:1397] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250624 14:19:25.126375 5274 raft_consensus.cc:357] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } }
I20250624 14:19:25.126961 5274 raft_consensus.cc:383] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:19:25.127161 5274 raft_consensus.cc:738] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4112eeddb57742e48ba4b705d3a0c0a7, State: Initialized, Role: FOLLOWER
I20250624 14:19:25.127835 5274 consensus_queue.cc:260] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } }
I20250624 14:19:25.128335 5274 raft_consensus.cc:397] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:19:25.128620 5274 raft_consensus.cc:491] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:19:25.128895 5274 raft_consensus.cc:3058] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:19:25.133704 5274 raft_consensus.cc:513] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } }
I20250624 14:19:25.134440 5274 leader_election.cc:304] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4112eeddb57742e48ba4b705d3a0c0a7; no voters:
I20250624 14:19:25.136234 5274 leader_election.cc:290] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 14:19:25.136611 5276 raft_consensus.cc:2802] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:19:25.139472 5276 raft_consensus.cc:695] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 1 LEADER]: Becoming Leader. State: Replica: 4112eeddb57742e48ba4b705d3a0c0a7, State: Running, Role: LEADER
I20250624 14:19:25.140744 5274 ts_tablet_manager.cc:1428] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Time spent starting tablet: real 0.032s user 0.028s sys 0.004s
I20250624 14:19:25.140501 5276 consensus_queue.cc:237] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } }
I20250624 14:19:25.152985 4778 catalog_manager.cc:5582] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2). New cstate: current_term: 1 leader_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } health_report { overall_health: HEALTHY } } }
I20250624 14:19:25.377897 2104 test_util.cc:276] Using random seed: 66640204
I20250624 14:19:25.400316 4776 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37978:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 14:19:25.430114 5168 tablet_service.cc:1468] Processing CreateTablet for tablet 8f2ce2bd8d124b8a858207ff7ba6e1db (DEFAULT_TABLE table=TestTable2 [id=d416107cfc6a4299acb39791b62918ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:19:25.431690 5168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2ce2bd8d124b8a858207ff7ba6e1db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:19:25.451637 5295 tablet_bootstrap.cc:492] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Bootstrap starting.
I20250624 14:19:25.457715 5295 tablet_bootstrap.cc:654] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Neither blocks nor log segments found. Creating new log.
I20250624 14:19:25.459486 5295 log.cc:826] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:25.464072 5295 tablet_bootstrap.cc:492] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: No bootstrap required, opened a new log
I20250624 14:19:25.464612 5295 ts_tablet_manager.cc:1397] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20250624 14:19:25.482277 5295 raft_consensus.cc:357] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } }
I20250624 14:19:25.482870 5295 raft_consensus.cc:383] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:19:25.483121 5295 raft_consensus.cc:738] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60919fec2a7d41739f65174c4e92014c, State: Initialized, Role: FOLLOWER
I20250624 14:19:25.483846 5295 consensus_queue.cc:260] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } }
I20250624 14:19:25.484365 5295 raft_consensus.cc:397] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:19:25.484643 5295 raft_consensus.cc:491] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:19:25.484951 5295 raft_consensus.cc:3058] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:19:25.489145 5295 raft_consensus.cc:513] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } }
I20250624 14:19:25.489864 5295 leader_election.cc:304] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60919fec2a7d41739f65174c4e92014c; no voters:
I20250624 14:19:25.492647 5295 leader_election.cc:290] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 14:19:25.493211 5297 raft_consensus.cc:2802] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:19:25.495704 5297 raft_consensus.cc:695] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 1 LEADER]: Becoming Leader. State: Replica: 60919fec2a7d41739f65174c4e92014c, State: Running, Role: LEADER
I20250624 14:19:25.496551 5233 heartbeater.cc:499] Master 127.2.14.62:41603 was elected leader, sending a full tablet report...
I20250624 14:19:25.496654 5297 consensus_queue.cc:237] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } }
I20250624 14:19:25.497591 5295 ts_tablet_manager.cc:1428] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Time spent starting tablet: real 0.033s user 0.025s sys 0.008s
I20250624 14:19:25.508715 4776 catalog_manager.cc:5582] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c reported cstate change: term changed from 0 to 1, leader changed from <none> to 60919fec2a7d41739f65174c4e92014c (127.2.14.3). New cstate: current_term: 1 leader_uuid: "60919fec2a7d41739f65174c4e92014c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } health_report { overall_health: HEALTHY } } }
I20250624 14:19:25.749181 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 4745
W20250624 14:19:25.825891 4966 heartbeater.cc:646] Failed to heartbeat to 127.2.14.62:41603 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.14.62:41603: connect: Connection refused (error 111)
W20250624 14:19:26.167905 5100 heartbeater.cc:646] Failed to heartbeat to 127.2.14.62:41603 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.14.62:41603: connect: Connection refused (error 111)
W20250624 14:19:26.526870 5233 heartbeater.cc:646] Failed to heartbeat to 127.2.14.62:41603 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.14.62:41603: connect: Connection refused (error 111)
I20250624 14:19:30.568972 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 4837
I20250624 14:19:30.594388 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 4970
I20250624 14:19:30.620942 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 5104
I20250624 14:19:30.651974 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:41603
--webserver_interface=127.2.14.62
--webserver_port=34921
--builtin_ntp_servers=127.2.14.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.14.62:41603 with env {}
W20250624 14:19:30.962303 5374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:30.962917 5374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:30.963371 5374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:30.994751 5374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 14:19:30.995081 5374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:30.995340 5374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 14:19:30.995595 5374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 14:19:31.032047 5374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.14.62:41603
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:41603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.14.62
--webserver_port=34921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:31.033396 5374 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:31.035113 5374 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:31.049777 5381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:31.050290 5380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:31.051952 5383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:31.052641 5374 server_base.cc:1048] running on GCE node
I20250624 14:19:32.240875 5374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:32.243600 5374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:32.245012 5374 hybrid_clock.cc:648] HybridClock initialized: now 1750774772244970 us; error 53 us; skew 500 ppm
I20250624 14:19:32.245905 5374 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:32.252686 5374 webserver.cc:469] Webserver started at http://127.2.14.62:34921/ using document root <none> and password file <none>
I20250624 14:19:32.253671 5374 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:32.253893 5374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:32.261979 5374 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250624 14:19:32.266697 5391 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:32.267855 5374 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250624 14:19:32.268190 5374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7db2648ab6e441c38549ea70be0355e3"
format_stamp: "Formatted at 2025-06-24 14:19:18 on dist-test-slave-hs17"
I20250624 14:19:32.270241 5374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:32.339468 5374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:32.341127 5374 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:32.341565 5374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:32.413308 5374 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.62:41603
I20250624 14:19:32.413390 5442 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.62:41603 every 8 connection(s)
I20250624 14:19:32.416163 5374 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 14:19:32.422581 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 5374
I20250624 14:19:32.424304 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.1:36801
--local_ip_for_outbound_sockets=127.2.14.1
--tserver_master_addrs=127.2.14.62:41603
--webserver_port=41237
--webserver_interface=127.2.14.1
--builtin_ntp_servers=127.2.14.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:19:32.426584 5443 sys_catalog.cc:263] Verifying existing consensus state
I20250624 14:19:32.431540 5443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Bootstrap starting.
I20250624 14:19:32.446624 5443 log.cc:826] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:32.511096 5443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:32.512015 5443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Bootstrap complete.
I20250624 14:19:32.533015 5443 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:32.535151 5443 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db2648ab6e441c38549ea70be0355e3, State: Initialized, Role: FOLLOWER
I20250624 14:19:32.536022 5443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:32.536542 5443 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:19:32.536800 5443 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:19:32.537096 5443 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 2 FOLLOWER]: Advancing to term 3
I20250624 14:19:32.542829 5443 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:32.543478 5443 leader_election.cc:304] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7db2648ab6e441c38549ea70be0355e3; no voters:
I20250624 14:19:32.545675 5443 leader_election.cc:290] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250624 14:19:32.546252 5447 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 3 FOLLOWER]: Leader election won for term 3
I20250624 14:19:32.549556 5447 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 3 LEADER]: Becoming Leader. State: Replica: 7db2648ab6e441c38549ea70be0355e3, State: Running, Role: LEADER
I20250624 14:19:32.550392 5443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 14:19:32.550468 5447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:32.563338 5448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7db2648ab6e441c38549ea70be0355e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } } }
I20250624 14:19:32.565797 5448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: This master's current role is: LEADER
I20250624 14:19:32.564970 5449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7db2648ab6e441c38549ea70be0355e3. Latest consensus state: current_term: 3 leader_uuid: "7db2648ab6e441c38549ea70be0355e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } } }
I20250624 14:19:32.568177 5449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: This master's current role is: LEADER
I20250624 14:19:32.577607 5453 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 14:19:32.592398 5453 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=76fe2eb3df674903a16acf89d22e43b5]
I20250624 14:19:32.594961 5453 catalog_manager.cc:671] Loaded metadata for table TestTable [id=b9ac46064bf740b3b27429663dd4db17]
I20250624 14:19:32.597239 5453 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=d416107cfc6a4299acb39791b62918ca]
I20250624 14:19:32.607998 5453 tablet_loader.cc:96] loaded metadata for tablet 678437be284e4cf29a9dbb50aaa3727d (table TestTable1 [id=76fe2eb3df674903a16acf89d22e43b5])
I20250624 14:19:32.609591 5453 tablet_loader.cc:96] loaded metadata for tablet 8f2ce2bd8d124b8a858207ff7ba6e1db (table TestTable2 [id=d416107cfc6a4299acb39791b62918ca])
I20250624 14:19:32.611287 5453 tablet_loader.cc:96] loaded metadata for tablet df16f2e37e1e471db624a209f2ade936 (table TestTable [id=b9ac46064bf740b3b27429663dd4db17])
I20250624 14:19:32.613564 5453 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 14:19:32.620662 5453 catalog_manager.cc:1261] Loaded cluster ID: db11613b64224d6a861fb54fb06c5787
I20250624 14:19:32.621134 5453 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 14:19:32.630932 5464 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250624 14:19:32.631997 5453 catalog_manager.cc:1506] Loading token signing keys...
I20250624 14:19:32.635740 5453 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Loaded TSK: 0
I20250624 14:19:32.636889 5453 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250624 14:19:32.797781 5445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:32.798308 5445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:32.798823 5445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:32.830029 5445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:32.830900 5445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.1
I20250624 14:19:32.866741 5445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.1:36801
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.14.1
--webserver_port=41237
--tserver_master_addrs=127.2.14.62:41603
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.1
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:32.868110 5445 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:32.869804 5445 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:32.888010 5471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:32.888118 5470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:32.891803 5445 server_base.cc:1048] running on GCE node
W20250624 14:19:32.890987 5473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:34.071314 5445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:34.074316 5445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:34.075763 5445 hybrid_clock.cc:648] HybridClock initialized: now 1750774774075702 us; error 80 us; skew 500 ppm
I20250624 14:19:34.076591 5445 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:34.089560 5445 webserver.cc:469] Webserver started at http://127.2.14.1:41237/ using document root <none> and password file <none>
I20250624 14:19:34.090653 5445 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:34.090915 5445 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:34.099963 5445 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.004s
I20250624 14:19:34.105815 5480 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:34.107015 5445 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20250624 14:19:34.107347 5445 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "55a21f9c8e154fcc85e4b33617f4aec7"
format_stamp: "Formatted at 2025-06-24 14:19:20 on dist-test-slave-hs17"
I20250624 14:19:34.109366 5445 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:34.158722 5445 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:34.160312 5445 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:34.160785 5445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:34.163983 5445 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:19:34.170171 5487 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250624 14:19:34.181588 5445 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 14:19:34.181869 5445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250624 14:19:34.182147 5445 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 14:19:34.187186 5445 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 14:19:34.187458 5445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.002s
I20250624 14:19:34.187963 5487 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Bootstrap starting.
I20250624 14:19:34.260509 5487 log.cc:826] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:34.393873 5445 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.1:36801
I20250624 14:19:34.394075 5594 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.1:36801 every 8 connection(s)
I20250624 14:19:34.397591 5445 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 14:19:34.404783 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 5445
I20250624 14:19:34.406654 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.2:39265
--local_ip_for_outbound_sockets=127.2.14.2
--tserver_master_addrs=127.2.14.62:41603
--webserver_port=45297
--webserver_interface=127.2.14.2
--builtin_ntp_servers=127.2.14.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:19:34.414759 5487 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:34.428445 5487 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Bootstrap complete.
I20250624 14:19:34.431038 5487 ts_tablet_manager.cc:1397] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Time spent bootstrapping tablet: real 0.244s user 0.174s sys 0.049s
I20250624 14:19:34.454669 5487 raft_consensus.cc:357] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } }
I20250624 14:19:34.457796 5487 raft_consensus.cc:738] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a21f9c8e154fcc85e4b33617f4aec7, State: Initialized, Role: FOLLOWER
I20250624 14:19:34.458665 5487 consensus_queue.cc:260] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } }
I20250624 14:19:34.459329 5487 raft_consensus.cc:397] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:19:34.459713 5487 raft_consensus.cc:491] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:19:34.460104 5487 raft_consensus.cc:3058] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 1 FOLLOWER]: Advancing to term 2
I20250624 14:19:34.460500 5595 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41603
I20250624 14:19:34.461025 5595 heartbeater.cc:461] Registering TS with master...
I20250624 14:19:34.462278 5595 heartbeater.cc:507] Master 127.2.14.62:41603 requested a full tablet report, sending...
I20250624 14:19:34.468384 5487 raft_consensus.cc:513] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } }
I20250624 14:19:34.469347 5487 leader_election.cc:304] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55a21f9c8e154fcc85e4b33617f4aec7; no voters:
I20250624 14:19:34.479401 5408 ts_manager.cc:194] Registered new tserver with Master: 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1:36801)
I20250624 14:19:34.489173 5408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.1:40499
I20250624 14:19:34.496878 5487 leader_election.cc:290] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 14:19:34.497681 5600 raft_consensus.cc:2802] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 14:19:34.506070 5600 raft_consensus.cc:695] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 LEADER]: Becoming Leader. State: Replica: 55a21f9c8e154fcc85e4b33617f4aec7, State: Running, Role: LEADER
I20250624 14:19:34.507241 5600 consensus_queue.cc:237] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } }
I20250624 14:19:34.509682 5487 ts_tablet_manager.cc:1428] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Time spent starting tablet: real 0.076s user 0.047s sys 0.022s
I20250624 14:19:34.510541 5595 heartbeater.cc:499] Master 127.2.14.62:41603 was elected leader, sending a full tablet report...
I20250624 14:19:34.515794 5408 catalog_manager.cc:5582] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 reported cstate change: term changed from 0 to 2, leader changed from <none> to 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1), VOTER 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1) added. New cstate: current_term: 2 leader_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } health_report { overall_health: HEALTHY } } }
W20250624 14:19:34.569046 5408 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet df16f2e37e1e471db624a209f2ade936 with cas_config_opid_index -1: no extra replica candidate found for tablet df16f2e37e1e471db624a209f2ade936 (table TestTable [id=b9ac46064bf740b3b27429663dd4db17]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250624 14:19:34.781139 5599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:34.781679 5599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:34.782186 5599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:34.833348 5599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:34.834860 5599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.2
I20250624 14:19:34.870353 5599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.2:39265
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.14.2
--webserver_port=45297
--tserver_master_addrs=127.2.14.62:41603
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.2
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:34.871652 5599 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:34.873291 5599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:34.889606 5615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:35.825119 5439 debug-util.cc:398] Leaking SignalData structure 0x7b080006f420 after lost signal to thread 5375
W20250624 14:19:35.825722 5439 debug-util.cc:398] Leaking SignalData structure 0x7b080006f400 after lost signal to thread 5442
W20250624 14:19:34.890476 5614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:34.891637 5599 server_base.cc:1048] running on GCE node
W20250624 14:19:34.892078 5617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:36.083998 5599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:36.086907 5599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:36.088420 5599 hybrid_clock.cc:648] HybridClock initialized: now 1750774776088384 us; error 56 us; skew 500 ppm
I20250624 14:19:36.089251 5599 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:36.096876 5599 webserver.cc:469] Webserver started at http://127.2.14.2:45297/ using document root <none> and password file <none>
I20250624 14:19:36.098124 5599 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:36.098378 5599 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:36.107146 5599 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.005s
I20250624 14:19:36.112733 5624 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:36.113960 5599 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250624 14:19:36.114301 5599 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4112eeddb57742e48ba4b705d3a0c0a7"
format_stamp: "Formatted at 2025-06-24 14:19:22 on dist-test-slave-hs17"
I20250624 14:19:36.116380 5599 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:36.184183 5599 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:36.185709 5599 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:36.186164 5599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:36.189219 5599 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:19:36.198108 5631 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250624 14:19:36.206566 5599 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 14:19:36.206826 5599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.002s sys 0.000s
I20250624 14:19:36.207114 5599 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 14:19:36.211988 5599 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 14:19:36.212263 5599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.003s
I20250624 14:19:36.212743 5631 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Bootstrap starting.
I20250624 14:19:36.287362 5631 log.cc:826] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:36.422611 5631 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:36.424142 5631 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Bootstrap complete.
I20250624 14:19:36.426347 5631 ts_tablet_manager.cc:1397] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Time spent bootstrapping tablet: real 0.214s user 0.173s sys 0.029s
I20250624 14:19:36.436803 5599 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.2:39265
I20250624 14:19:36.436981 5738 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.2:39265 every 8 connection(s)
I20250624 14:19:36.440939 5599 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 14:19:36.444375 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 5599
I20250624 14:19:36.446259 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.3:42843
--local_ip_for_outbound_sockets=127.2.14.3
--tserver_master_addrs=127.2.14.62:41603
--webserver_port=34735
--webserver_interface=127.2.14.3
--builtin_ntp_servers=127.2.14.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:19:36.464421 5631 raft_consensus.cc:357] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } }
I20250624 14:19:36.478770 5631 raft_consensus.cc:738] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4112eeddb57742e48ba4b705d3a0c0a7, State: Initialized, Role: FOLLOWER
I20250624 14:19:36.479988 5631 consensus_queue.cc:260] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } }
I20250624 14:19:36.480742 5631 raft_consensus.cc:397] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:19:36.481228 5631 raft_consensus.cc:491] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:19:36.481842 5631 raft_consensus.cc:3058] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 1 FOLLOWER]: Advancing to term 2
I20250624 14:19:36.490525 5631 raft_consensus.cc:513] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } }
I20250624 14:19:36.491609 5631 leader_election.cc:304] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4112eeddb57742e48ba4b705d3a0c0a7; no voters:
I20250624 14:19:36.493481 5739 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41603
I20250624 14:19:36.494156 5739 heartbeater.cc:461] Registering TS with master...
I20250624 14:19:36.495890 5739 heartbeater.cc:507] Master 127.2.14.62:41603 requested a full tablet report, sending...
I20250624 14:19:36.502795 5408 ts_manager.cc:194] Registered new tserver with Master: 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2:39265)
I20250624 14:19:36.504590 5744 raft_consensus.cc:2802] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 14:19:36.504670 5631 leader_election.cc:290] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 14:19:36.510176 5408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.2:53055
I20250624 14:19:36.512137 5744 raft_consensus.cc:695] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 LEADER]: Becoming Leader. State: Replica: 4112eeddb57742e48ba4b705d3a0c0a7, State: Running, Role: LEADER
I20250624 14:19:36.513188 5744 consensus_queue.cc:237] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } }
I20250624 14:19:36.514534 5739 heartbeater.cc:499] Master 127.2.14.62:41603 was elected leader, sending a full tablet report...
I20250624 14:19:36.517859 5631 ts_tablet_manager.cc:1428] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Time spent starting tablet: real 0.091s user 0.043s sys 0.012s
I20250624 14:19:36.531055 5408 catalog_manager.cc:5582] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 reported cstate change: term changed from 0 to 2, leader changed from <none> to 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2), VOTER 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2) added. New cstate: current_term: 2 leader_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } health_report { overall_health: HEALTHY } } }
I20250624 14:19:36.581108 5694 consensus_queue.cc:237] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: NON_VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: true } }
I20250624 14:19:36.585747 5746 raft_consensus.cc:2953] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: NON_VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: true } } }
I20250624 14:19:36.598055 5393 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 678437be284e4cf29a9dbb50aaa3727d with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250624 14:19:36.603073 5408 catalog_manager.cc:5582] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 reported cstate change: config changed from index -1 to 8, NON_VOTER 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1) added. New cstate: current_term: 2 leader_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: NON_VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250624 14:19:36.617913 5626 consensus_peers.cc:487] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 -> Peer 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1:36801): Couldn't send request to peer 55a21f9c8e154fcc85e4b33617f4aec7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 678437be284e4cf29a9dbb50aaa3727d. This is attempt 1: this message will repeat every 5th retry.
I20250624 14:19:36.880168 5550 consensus_queue.cc:237] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: NON_VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: true } }
I20250624 14:19:36.906206 5758 raft_consensus.cc:2953] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: NON_VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: true } } }
I20250624 14:19:36.920781 5393 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet df16f2e37e1e471db624a209f2ade936 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
W20250624 14:19:36.923730 5482 consensus_peers.cc:487] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 -> Peer 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2:39265): Couldn't send request to peer 4112eeddb57742e48ba4b705d3a0c0a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df16f2e37e1e471db624a209f2ade936. This is attempt 1: this message will repeat every 5th retry.
I20250624 14:19:36.926810 5408 catalog_manager.cc:5582] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 reported cstate change: config changed from index -1 to 9, NON_VOTER 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2) added. New cstate: current_term: 2 leader_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: NON_VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250624 14:19:36.958813 5743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:36.959331 5743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:36.959900 5743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:36.991632 5743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:36.992509 5743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.3
I20250624 14:19:37.028043 5743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.3:42843
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.14.3
--webserver_port=34735
--tserver_master_addrs=127.2.14.62:41603
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.3
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:37.029361 5743 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:37.031020 5743 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:37.049671 5765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:37.205233 5772 ts_tablet_manager.cc:927] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: Initiating tablet copy from peer 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2:39265)
I20250624 14:19:37.220563 5772 tablet_copy_client.cc:323] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: tablet copy: Beginning tablet copy session from remote peer at address 127.2.14.2:39265
I20250624 14:19:37.223453 5714 tablet_copy_service.cc:140] P 4112eeddb57742e48ba4b705d3a0c0a7: Received BeginTabletCopySession request for tablet 678437be284e4cf29a9dbb50aaa3727d from peer 55a21f9c8e154fcc85e4b33617f4aec7 ({username='slave'} at 127.2.14.1:35485)
I20250624 14:19:37.224313 5714 tablet_copy_service.cc:161] P 4112eeddb57742e48ba4b705d3a0c0a7: Beginning new tablet copy session on tablet 678437be284e4cf29a9dbb50aaa3727d from peer 55a21f9c8e154fcc85e4b33617f4aec7 at {username='slave'} at 127.2.14.1:35485: session id = 55a21f9c8e154fcc85e4b33617f4aec7-678437be284e4cf29a9dbb50aaa3727d
I20250624 14:19:37.234962 5714 tablet_copy_source_session.cc:215] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 14:19:37.241281 5772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678437be284e4cf29a9dbb50aaa3727d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:19:37.264544 5772 tablet_copy_client.cc:806] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: tablet copy: Starting download of 0 data blocks...
I20250624 14:19:37.265540 5772 tablet_copy_client.cc:670] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: tablet copy: Starting download of 1 WAL segments...
I20250624 14:19:37.272181 5772 tablet_copy_client.cc:538] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 14:19:37.283814 5772 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: Bootstrap starting.
I20250624 14:19:37.341614 5775 ts_tablet_manager.cc:927] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: Initiating tablet copy from peer 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1:36801)
I20250624 14:19:37.357635 5775 tablet_copy_client.cc:323] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: tablet copy: Beginning tablet copy session from remote peer at address 127.2.14.1:36801
I20250624 14:19:37.360574 5570 tablet_copy_service.cc:140] P 55a21f9c8e154fcc85e4b33617f4aec7: Received BeginTabletCopySession request for tablet df16f2e37e1e471db624a209f2ade936 from peer 4112eeddb57742e48ba4b705d3a0c0a7 ({username='slave'} at 127.2.14.2:33745)
I20250624 14:19:37.361423 5570 tablet_copy_service.cc:161] P 55a21f9c8e154fcc85e4b33617f4aec7: Beginning new tablet copy session on tablet df16f2e37e1e471db624a209f2ade936 from peer 4112eeddb57742e48ba4b705d3a0c0a7 at {username='slave'} at 127.2.14.2:33745: session id = 4112eeddb57742e48ba4b705d3a0c0a7-df16f2e37e1e471db624a209f2ade936
I20250624 14:19:37.371861 5570 tablet_copy_source_session.cc:215] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 14:19:37.377537 5775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df16f2e37e1e471db624a209f2ade936. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:19:37.503716 5775 tablet_copy_client.cc:806] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: tablet copy: Starting download of 0 data blocks...
I20250624 14:19:37.504675 5775 tablet_copy_client.cc:670] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: tablet copy: Starting download of 1 WAL segments...
I20250624 14:19:37.517729 5775 tablet_copy_client.cc:538] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 14:19:37.538564 5775 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: Bootstrap starting.
I20250624 14:19:37.822883 5772 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:37.830699 5772 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: Bootstrap complete.
I20250624 14:19:37.837363 5772 ts_tablet_manager.cc:1397] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: Time spent bootstrapping tablet: real 0.554s user 0.154s sys 0.013s
I20250624 14:19:37.840035 5772 raft_consensus.cc:357] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: NON_VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: true } }
I20250624 14:19:37.840797 5772 raft_consensus.cc:738] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 55a21f9c8e154fcc85e4b33617f4aec7, State: Initialized, Role: LEARNER
I20250624 14:19:37.847997 5772 consensus_queue.cc:260] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: NON_VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: true } }
I20250624 14:19:37.857076 5772 ts_tablet_manager.cc:1428] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20250624 14:19:37.858748 5714 tablet_copy_service.cc:342] P 4112eeddb57742e48ba4b705d3a0c0a7: Request end of tablet copy session 55a21f9c8e154fcc85e4b33617f4aec7-678437be284e4cf29a9dbb50aaa3727d received from {username='slave'} at 127.2.14.1:35485
I20250624 14:19:37.859217 5714 tablet_copy_service.cc:434] P 4112eeddb57742e48ba4b705d3a0c0a7: ending tablet copy session 55a21f9c8e154fcc85e4b33617f4aec7-678437be284e4cf29a9dbb50aaa3727d on tablet 678437be284e4cf29a9dbb50aaa3727d with peer 55a21f9c8e154fcc85e4b33617f4aec7
I20250624 14:19:37.972081 5775 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:37.973373 5775 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: Bootstrap complete.
I20250624 14:19:37.974237 5775 ts_tablet_manager.cc:1397] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: Time spent bootstrapping tablet: real 0.436s user 0.160s sys 0.016s
I20250624 14:19:37.977555 5775 raft_consensus.cc:357] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: NON_VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: true } }
I20250624 14:19:37.978508 5775 raft_consensus.cc:738] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4112eeddb57742e48ba4b705d3a0c0a7, State: Initialized, Role: LEARNER
I20250624 14:19:37.979300 5775 consensus_queue.cc:260] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: NON_VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: true } }
I20250624 14:19:37.984922 5775 ts_tablet_manager.cc:1428] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250624 14:19:37.986764 5570 tablet_copy_service.cc:342] P 55a21f9c8e154fcc85e4b33617f4aec7: Request end of tablet copy session 4112eeddb57742e48ba4b705d3a0c0a7-df16f2e37e1e471db624a209f2ade936 received from {username='slave'} at 127.2.14.2:33745
I20250624 14:19:37.987246 5570 tablet_copy_service.cc:434] P 55a21f9c8e154fcc85e4b33617f4aec7: ending tablet copy session 4112eeddb57742e48ba4b705d3a0c0a7-df16f2e37e1e471db624a209f2ade936 on tablet df16f2e37e1e471db624a209f2ade936 with peer 4112eeddb57742e48ba4b705d3a0c0a7
I20250624 14:19:38.282260 5550 raft_consensus.cc:1215] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20250624 14:19:38.350445 5694 raft_consensus.cc:1215] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20250624 14:19:37.051549 5768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:38.511783 5767 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1462 milliseconds
I20250624 14:19:38.512045 5743 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250624 14:19:37.049819 5766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:38.517539 5743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:38.520768 5743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:38.522348 5743 hybrid_clock.cc:648] HybridClock initialized: now 1750774778522303 us; error 42 us; skew 500 ppm
I20250624 14:19:38.523535 5743 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:38.531960 5743 webserver.cc:469] Webserver started at http://127.2.14.3:34735/ using document root <none> and password file <none>
I20250624 14:19:38.533314 5743 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:38.533631 5743 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:38.545817 5743 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.005s sys 0.004s
I20250624 14:19:38.552381 5787 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:38.553757 5743 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20250624 14:19:38.554214 5743 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "60919fec2a7d41739f65174c4e92014c"
format_stamp: "Formatted at 2025-06-24 14:19:24 on dist-test-slave-hs17"
I20250624 14:19:38.557379 5743 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:38.658128 5778 raft_consensus.cc:1062] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: attempting to promote NON_VOTER 55a21f9c8e154fcc85e4b33617f4aec7 to VOTER
I20250624 14:19:38.660384 5778 consensus_queue.cc:237] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } }
I20250624 14:19:38.665500 5550 raft_consensus.cc:1273] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 LEARNER]: Refusing update from remote peer 4112eeddb57742e48ba4b705d3a0c0a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250624 14:19:38.666643 5778 consensus_queue.cc:1035] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250624 14:19:38.673313 5781 raft_consensus.cc:2953] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } }
I20250624 14:19:38.676353 5743 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:38.678336 5743 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:38.678987 5743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:38.678354 5550 raft_consensus.cc:2953] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } }
I20250624 14:19:38.685251 5408 catalog_manager.cc:5582] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 reported cstate change: config changed from index 8 to 9, 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250624 14:19:38.698114 5743 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:19:38.711146 5800 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250624 14:19:38.723533 5743 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 14:19:38.723866 5743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.002s sys 0.001s
I20250624 14:19:38.724251 5743 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 14:19:38.730392 5743 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 14:19:38.730675 5743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.001s sys 0.003s
I20250624 14:19:38.731106 5800 tablet_bootstrap.cc:492] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Bootstrap starting.
I20250624 14:19:38.803081 5800 log.cc:826] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:38.835541 5394 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 678437be284e4cf29a9dbb50aaa3727d with cas_config_opid_index 8: aborting the task: latest config opid_index 9; task opid_index 8
I20250624 14:19:38.906505 5800 tablet_bootstrap.cc:492] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:38.908134 5800 tablet_bootstrap.cc:492] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Bootstrap complete.
I20250624 14:19:38.909963 5800 ts_tablet_manager.cc:1397] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Time spent bootstrapping tablet: real 0.179s user 0.127s sys 0.048s
I20250624 14:19:38.920547 5899 raft_consensus.cc:1062] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: attempting to promote NON_VOTER 4112eeddb57742e48ba4b705d3a0c0a7 to VOTER
I20250624 14:19:38.922480 5899 consensus_queue.cc:237] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } }
I20250624 14:19:38.934768 5694 raft_consensus.cc:1273] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 LEARNER]: Refusing update from remote peer 55a21f9c8e154fcc85e4b33617f4aec7: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250624 14:19:38.936614 5901 consensus_queue.cc:1035] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250624 14:19:38.936125 5800 raft_consensus.cc:357] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } }
I20250624 14:19:38.939795 5800 raft_consensus.cc:738] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60919fec2a7d41739f65174c4e92014c, State: Initialized, Role: FOLLOWER
I20250624 14:19:38.940976 5743 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.3:42843
I20250624 14:19:38.941157 5910 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.3:42843 every 8 connection(s)
I20250624 14:19:38.941112 5800 consensus_queue.cc:260] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } }
I20250624 14:19:38.941937 5800 raft_consensus.cc:397] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:19:38.942291 5800 raft_consensus.cc:491] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:19:38.942754 5800 raft_consensus.cc:3058] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 1 FOLLOWER]: Advancing to term 2
I20250624 14:19:38.944607 5743 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 14:19:38.946111 5899 raft_consensus.cc:2953] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } }
I20250624 14:19:38.948832 5694 raft_consensus.cc:2953] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } }
I20250624 14:19:38.951618 5800 raft_consensus.cc:513] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } }
I20250624 14:19:38.952811 5800 leader_election.cc:304] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60919fec2a7d41739f65174c4e92014c; no voters:
I20250624 14:19:38.954914 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 5743
I20250624 14:19:38.970115 5408 catalog_manager.cc:5582] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 reported cstate change: config changed from index 9 to 10, 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } }
I20250624 14:19:38.983958 5800 leader_election.cc:290] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 14:19:38.986726 5913 raft_consensus.cc:2802] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Leader election won for term 2
I20250624 14:19:39.011726 5913 raft_consensus.cc:695] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 2 LEADER]: Becoming Leader. State: Replica: 60919fec2a7d41739f65174c4e92014c, State: Running, Role: LEADER
I20250624 14:19:39.012252 5800 ts_tablet_manager.cc:1428] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Time spent starting tablet: real 0.102s user 0.036s sys 0.019s
I20250624 14:19:39.012728 5913 consensus_queue.cc:237] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } }
I20250624 14:19:39.013670 5912 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41603
I20250624 14:19:39.014127 5912 heartbeater.cc:461] Registering TS with master...
I20250624 14:19:39.015139 5912 heartbeater.cc:507] Master 127.2.14.62:41603 requested a full tablet report, sending...
I20250624 14:19:39.019843 5406 ts_manager.cc:194] Registered new tserver with Master: 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843)
I20250624 14:19:39.020901 5406 catalog_manager.cc:5582] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "60919fec2a7d41739f65174c4e92014c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } health_report { overall_health: HEALTHY } } }
I20250624 14:19:39.022539 2104 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 14:19:39.028407 2104 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 14:19:39.028931 5406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.3:36221
I20250624 14:19:39.032737 5912 heartbeater.cc:499] Master 127.2.14.62:41603 was elected leader, sending a full tablet report...
W20250624 14:19:39.033010 2104 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250624 14:19:39.043898 5550 consensus_queue.cc:237] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: NON_VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: true } }
I20250624 14:19:39.048669 5694 raft_consensus.cc:1273] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Refusing update from remote peer 55a21f9c8e154fcc85e4b33617f4aec7: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250624 14:19:39.050736 5899 consensus_queue.cc:1035] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250624 14:19:39.056483 5758 raft_consensus.cc:2953] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 60919fec2a7d41739f65174c4e92014c (127.2.14.3) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: NON_VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: true } } }
I20250624 14:19:39.058326 5694 raft_consensus.cc:2953] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 60919fec2a7d41739f65174c4e92014c (127.2.14.3) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: NON_VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: true } } }
W20250624 14:19:39.062574 5484 consensus_peers.cc:487] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 -> Peer 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843): Couldn't send request to peer 60919fec2a7d41739f65174c4e92014c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df16f2e37e1e471db624a209f2ade936. This is attempt 1: this message will repeat every 5th retry.
I20250624 14:19:39.065162 5393 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet df16f2e37e1e471db624a209f2ade936 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 2)
I20250624 14:19:39.068648 5406 catalog_manager.cc:5582] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 reported cstate change: config changed from index 10 to 11, NON_VOTER 60919fec2a7d41739f65174c4e92014c (127.2.14.3) added. New cstate: current_term: 2 leader_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: NON_VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250624 14:19:39.075222 5694 consensus_queue.cc:237] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: NON_VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: true } }
I20250624 14:19:39.080734 5550 raft_consensus.cc:1273] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Refusing update from remote peer 4112eeddb57742e48ba4b705d3a0c0a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250624 14:19:39.082212 5779 consensus_queue.cc:1035] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250624 14:19:39.088065 5781 raft_consensus.cc:2953] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 60919fec2a7d41739f65174c4e92014c (127.2.14.3) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: NON_VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: true } } }
I20250624 14:19:39.089780 5550 raft_consensus.cc:2953] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 60919fec2a7d41739f65174c4e92014c (127.2.14.3) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: NON_VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: true } } }
I20250624 14:19:39.098640 5393 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 678437be284e4cf29a9dbb50aaa3727d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
W20250624 14:19:39.098899 5628 consensus_peers.cc:487] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 -> Peer 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843): Couldn't send request to peer 60919fec2a7d41739f65174c4e92014c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 678437be284e4cf29a9dbb50aaa3727d. This is attempt 1: this message will repeat every 5th retry.
I20250624 14:19:39.100301 5407 catalog_manager.cc:5582] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 reported cstate change: config changed from index 9 to 10, NON_VOTER 60919fec2a7d41739f65174c4e92014c (127.2.14.3) added. New cstate: current_term: 2 leader_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: NON_VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: true } } }
I20250624 14:19:39.145948 5394 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet df16f2e37e1e471db624a209f2ade936 with cas_config_opid_index 9: aborting the task: latest config opid_index 11; task opid_index 9
I20250624 14:19:39.537662 5926 ts_tablet_manager.cc:927] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: Initiating tablet copy from peer 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1:36801)
I20250624 14:19:39.539513 5926 tablet_copy_client.cc:323] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: tablet copy: Beginning tablet copy session from remote peer at address 127.2.14.1:36801
I20250624 14:19:39.549594 5570 tablet_copy_service.cc:140] P 55a21f9c8e154fcc85e4b33617f4aec7: Received BeginTabletCopySession request for tablet df16f2e37e1e471db624a209f2ade936 from peer 60919fec2a7d41739f65174c4e92014c ({username='slave'} at 127.2.14.3:39577)
I20250624 14:19:39.550271 5570 tablet_copy_service.cc:161] P 55a21f9c8e154fcc85e4b33617f4aec7: Beginning new tablet copy session on tablet df16f2e37e1e471db624a209f2ade936 from peer 60919fec2a7d41739f65174c4e92014c at {username='slave'} at 127.2.14.3:39577: session id = 60919fec2a7d41739f65174c4e92014c-df16f2e37e1e471db624a209f2ade936
I20250624 14:19:39.555673 5570 tablet_copy_source_session.cc:215] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 14:19:39.559161 5926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df16f2e37e1e471db624a209f2ade936. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:19:39.560272 5930 ts_tablet_manager.cc:927] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: Initiating tablet copy from peer 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2:39265)
I20250624 14:19:39.562842 5930 tablet_copy_client.cc:323] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: tablet copy: Beginning tablet copy session from remote peer at address 127.2.14.2:39265
I20250624 14:19:39.574718 5926 tablet_copy_client.cc:806] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: tablet copy: Starting download of 0 data blocks...
I20250624 14:19:39.575273 5926 tablet_copy_client.cc:670] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: tablet copy: Starting download of 1 WAL segments...
I20250624 14:19:39.575801 5714 tablet_copy_service.cc:140] P 4112eeddb57742e48ba4b705d3a0c0a7: Received BeginTabletCopySession request for tablet 678437be284e4cf29a9dbb50aaa3727d from peer 60919fec2a7d41739f65174c4e92014c ({username='slave'} at 127.2.14.3:47179)
I20250624 14:19:39.576206 5714 tablet_copy_service.cc:161] P 4112eeddb57742e48ba4b705d3a0c0a7: Beginning new tablet copy session on tablet 678437be284e4cf29a9dbb50aaa3727d from peer 60919fec2a7d41739f65174c4e92014c at {username='slave'} at 127.2.14.3:47179: session id = 60919fec2a7d41739f65174c4e92014c-678437be284e4cf29a9dbb50aaa3727d
I20250624 14:19:39.579551 5926 tablet_copy_client.cc:538] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 14:19:39.582335 5714 tablet_copy_source_session.cc:215] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 14:19:39.585377 5930 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678437be284e4cf29a9dbb50aaa3727d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:19:39.588163 5926 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: Bootstrap starting.
I20250624 14:19:39.601708 5930 tablet_copy_client.cc:806] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: tablet copy: Starting download of 0 data blocks...
I20250624 14:19:39.602257 5930 tablet_copy_client.cc:670] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: tablet copy: Starting download of 1 WAL segments...
I20250624 14:19:39.606235 5930 tablet_copy_client.cc:538] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 14:19:39.614964 5930 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: Bootstrap starting.
I20250624 14:19:39.721810 5926 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:39.722623 5926 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: Bootstrap complete.
I20250624 14:19:39.723214 5926 ts_tablet_manager.cc:1397] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: Time spent bootstrapping tablet: real 0.135s user 0.114s sys 0.017s
I20250624 14:19:39.725768 5926 raft_consensus.cc:357] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: NON_VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: true } }
I20250624 14:19:39.726465 5926 raft_consensus.cc:738] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 60919fec2a7d41739f65174c4e92014c, State: Initialized, Role: LEARNER
I20250624 14:19:39.727028 5926 consensus_queue.cc:260] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: NON_VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: true } }
I20250624 14:19:39.730211 5926 ts_tablet_manager.cc:1428] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20250624 14:19:39.732077 5930 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:39.732144 5570 tablet_copy_service.cc:342] P 55a21f9c8e154fcc85e4b33617f4aec7: Request end of tablet copy session 60919fec2a7d41739f65174c4e92014c-df16f2e37e1e471db624a209f2ade936 received from {username='slave'} at 127.2.14.3:39577
I20250624 14:19:39.732511 5570 tablet_copy_service.cc:434] P 55a21f9c8e154fcc85e4b33617f4aec7: ending tablet copy session 60919fec2a7d41739f65174c4e92014c-df16f2e37e1e471db624a209f2ade936 on tablet df16f2e37e1e471db624a209f2ade936 with peer 60919fec2a7d41739f65174c4e92014c
I20250624 14:19:39.733019 5930 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: Bootstrap complete.
I20250624 14:19:39.733631 5930 ts_tablet_manager.cc:1397] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: Time spent bootstrapping tablet: real 0.119s user 0.100s sys 0.021s
I20250624 14:19:39.735680 5930 raft_consensus.cc:357] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: NON_VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: true } }
I20250624 14:19:39.736541 5930 raft_consensus.cc:738] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 60919fec2a7d41739f65174c4e92014c, State: Initialized, Role: LEARNER
I20250624 14:19:39.737080 5930 consensus_queue.cc:260] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: NON_VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: true } }
I20250624 14:19:39.738632 5930 ts_tablet_manager.cc:1428] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250624 14:19:39.740106 5714 tablet_copy_service.cc:342] P 4112eeddb57742e48ba4b705d3a0c0a7: Request end of tablet copy session 60919fec2a7d41739f65174c4e92014c-678437be284e4cf29a9dbb50aaa3727d received from {username='slave'} at 127.2.14.3:47179
I20250624 14:19:39.740464 5714 tablet_copy_service.cc:434] P 4112eeddb57742e48ba4b705d3a0c0a7: ending tablet copy session 60919fec2a7d41739f65174c4e92014c-678437be284e4cf29a9dbb50aaa3727d on tablet 678437be284e4cf29a9dbb50aaa3727d with peer 60919fec2a7d41739f65174c4e92014c
I20250624 14:19:40.037513 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 55a21f9c8e154fcc85e4b33617f4aec7 to finish bootstrapping
I20250624 14:19:40.054858 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4112eeddb57742e48ba4b705d3a0c0a7 to finish bootstrapping
I20250624 14:19:40.066143 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 60919fec2a7d41739f65174c4e92014c to finish bootstrapping
I20250624 14:19:40.127199 5863 raft_consensus.cc:1215] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250624 14:19:40.164011 5863 raft_consensus.cc:1215] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250624 14:19:40.352210 5674 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 14:19:40.356176 5530 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 14:19:40.364915 5843 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
7db2648ab6e441c38549ea70be0355e3 | 127.2.14.62:41603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.14.20:41307 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
4112eeddb57742e48ba4b705d3a0c0a7 | 127.2.14.2:39265 | HEALTHY | <none> | 1 | 0
55a21f9c8e154fcc85e4b33617f4aec7 | 127.2.14.1:36801 | HEALTHY | <none> | 1 | 0
60919fec2a7d41739f65174c4e92014c | 127.2.14.3:42843 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.2.14.1 | experimental | 127.2.14.1:36801
local_ip_for_outbound_sockets | 127.2.14.2 | experimental | 127.2.14.2:39265
local_ip_for_outbound_sockets | 127.2.14.3 | experimental | 127.2.14.3:42843
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.14.1:36801
server_dump_info_path | /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.14.2:39265
server_dump_info_path | /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.14.3:42843
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.14.20:41307 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250624 14:19:40.645072 5945 raft_consensus.cc:1062] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: attempting to promote NON_VOTER 60919fec2a7d41739f65174c4e92014c to VOTER
I20250624 14:19:40.645200 2104 log_verifier.cc:126] Checking tablet 678437be284e4cf29a9dbb50aaa3727d
I20250624 14:19:40.646507 5945 consensus_queue.cc:237] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:40.653312 5863 raft_consensus.cc:1273] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c [term 2 LEARNER]: Refusing update from remote peer 55a21f9c8e154fcc85e4b33617f4aec7: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250624 14:19:40.653537 5694 raft_consensus.cc:1273] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Refusing update from remote peer 55a21f9c8e154fcc85e4b33617f4aec7: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250624 14:19:40.657976 5969 consensus_queue.cc:1035] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250624 14:19:40.659834 5945 consensus_queue.cc:1035] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250624 14:19:40.674784 5969 raft_consensus.cc:2953] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 60919fec2a7d41739f65174c4e92014c (127.2.14.3) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } } }
I20250624 14:19:40.681834 5863 raft_consensus.cc:2953] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 60919fec2a7d41739f65174c4e92014c (127.2.14.3) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } } }
I20250624 14:19:40.684264 5694 raft_consensus.cc:2953] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 60919fec2a7d41739f65174c4e92014c (127.2.14.3) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } } }
I20250624 14:19:40.686311 5407 catalog_manager.cc:5582] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 reported cstate change: config changed from index 11 to 12, 60919fec2a7d41739f65174c4e92014c (127.2.14.3) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250624 14:19:40.756319 5948 raft_consensus.cc:1062] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: attempting to promote NON_VOTER 60919fec2a7d41739f65174c4e92014c to VOTER
I20250624 14:19:40.757751 5948 consensus_queue.cc:237] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:40.763159 5863 raft_consensus.cc:1273] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c [term 2 LEARNER]: Refusing update from remote peer 4112eeddb57742e48ba4b705d3a0c0a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250624 14:19:40.764592 5550 raft_consensus.cc:1273] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Refusing update from remote peer 4112eeddb57742e48ba4b705d3a0c0a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250624 14:19:40.764787 5978 consensus_queue.cc:1035] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250624 14:19:40.766350 5978 consensus_queue.cc:1035] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250624 14:19:40.783416 5978 raft_consensus.cc:2953] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 60919fec2a7d41739f65174c4e92014c (127.2.14.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } } }
I20250624 14:19:40.786285 2104 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 678437be284e4cf29a9dbb50aaa3727d
I20250624 14:19:40.786607 2104 log_verifier.cc:126] Checking tablet 8f2ce2bd8d124b8a858207ff7ba6e1db
I20250624 14:19:40.785537 5863 raft_consensus.cc:2953] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 60919fec2a7d41739f65174c4e92014c (127.2.14.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } } }
I20250624 14:19:40.791370 5550 raft_consensus.cc:2953] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 60919fec2a7d41739f65174c4e92014c (127.2.14.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } } }
I20250624 14:19:40.797505 5408 catalog_manager.cc:5582] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 reported cstate change: config changed from index 10 to 11, 60919fec2a7d41739f65174c4e92014c (127.2.14.3) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250624 14:19:40.824191 2104 log_verifier.cc:177] Verified matching terms for 8 ops in tablet 8f2ce2bd8d124b8a858207ff7ba6e1db
I20250624 14:19:40.824409 2104 log_verifier.cc:126] Checking tablet df16f2e37e1e471db624a209f2ade936
I20250624 14:19:40.909251 2104 log_verifier.cc:177] Verified matching terms for 12 ops in tablet df16f2e37e1e471db624a209f2ade936
I20250624 14:19:40.909636 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 5374
I20250624 14:19:40.944741 2104 minidump.cc:252] Setting minidump size limit to 20M
I20250624 14:19:40.946123 2104 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:40.947466 2104 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:40.960703 5985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:40.960789 5984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:41.176663 5987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:41.176750 2104 server_base.cc:1048] running on GCE node
I20250624 14:19:41.178102 2104 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250624 14:19:41.178309 2104 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250624 14:19:41.178447 2104 hybrid_clock.cc:648] HybridClock initialized: now 1750774781178430 us; error 0 us; skew 500 ppm
I20250624 14:19:41.179071 2104 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:41.182094 2104 webserver.cc:469] Webserver started at http://0.0.0.0:43897/ using document root <none> and password file <none>
I20250624 14:19:41.182893 2104 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:41.183073 2104 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:41.188266 2104 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20250624 14:19:41.191900 5992 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:41.192826 2104 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250624 14:19:41.193136 2104 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7db2648ab6e441c38549ea70be0355e3"
format_stamp: "Formatted at 2025-06-24 14:19:18 on dist-test-slave-hs17"
I20250624 14:19:41.194965 2104 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:41.213981 2104 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:41.215468 2104 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:41.215979 2104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:41.225205 2104 sys_catalog.cc:263] Verifying existing consensus state
W20250624 14:19:41.228911 2104 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.2.14.62:41603 exists but no master address supplied!
I20250624 14:19:41.231119 2104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Bootstrap starting.
I20250624 14:19:41.276090 2104 log.cc:826] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:41.342924 2104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:41.343896 2104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Bootstrap complete.
I20250624 14:19:41.358156 2104 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:41.358793 2104 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db2648ab6e441c38549ea70be0355e3, State: Initialized, Role: FOLLOWER
I20250624 14:19:41.359679 2104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:41.360265 2104 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:19:41.360646 2104 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:19:41.361065 2104 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 3 FOLLOWER]: Advancing to term 4
I20250624 14:19:41.367123 2104 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:41.367898 2104 leader_election.cc:304] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7db2648ab6e441c38549ea70be0355e3; no voters:
I20250624 14:19:41.369077 2104 leader_election.cc:290] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250624 14:19:41.369341 5999 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 4 FOLLOWER]: Leader election won for term 4
I20250624 14:19:41.371371 5999 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 4 LEADER]: Becoming Leader. State: Replica: 7db2648ab6e441c38549ea70be0355e3, State: Running, Role: LEADER
I20250624 14:19:41.372221 5999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:41.378896 6000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "7db2648ab6e441c38549ea70be0355e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } } }
I20250624 14:19:41.379433 6000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: This master's current role is: LEADER
I20250624 14:19:41.380506 6001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7db2648ab6e441c38549ea70be0355e3. Latest consensus state: current_term: 4 leader_uuid: "7db2648ab6e441c38549ea70be0355e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } } }
I20250624 14:19:41.381069 6001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: This master's current role is: LEADER
I20250624 14:19:41.415971 2104 tablet_replica.cc:331] stopping tablet replica
I20250624 14:19:41.416664 2104 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 4 LEADER]: Raft consensus shutting down.
I20250624 14:19:41.417228 2104 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250624 14:19:41.420661 2104 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250624 14:19:41.421362 2104 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250624 14:19:41.453866 2104 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250624 14:19:41.828451 5595 heartbeater.cc:646] Failed to heartbeat to 127.2.14.62:41603 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.14.62:41603: connect: Connection refused (error 111)
W20250624 14:19:41.831460 5912 heartbeater.cc:646] Failed to heartbeat to 127.2.14.62:41603 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.14.62:41603: connect: Connection refused (error 111)
W20250624 14:19:41.834301 5739 heartbeater.cc:646] Failed to heartbeat to 127.2.14.62:41603 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.14.62:41603: connect: Connection refused (error 111)
I20250624 14:19:46.677197 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 5445
I20250624 14:19:46.705374 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 5599
I20250624 14:19:46.735291 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 5743
I20250624 14:19:46.770532 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:41603
--webserver_interface=127.2.14.62
--webserver_port=34921
--builtin_ntp_servers=127.2.14.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.14.62:41603 with env {}
W20250624 14:19:47.084216 6073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:47.084841 6073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:47.085316 6073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:47.117036 6073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 14:19:47.117391 6073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:47.117653 6073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 14:19:47.117906 6073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 14:19:47.153770 6073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.14.62:41603
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:41603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.14.62
--webserver_port=34921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:47.155164 6073 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:47.156862 6073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:47.173321 6080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:47.173316 6079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:47.173303 6082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:47.175460 6073 server_base.cc:1048] running on GCE node
I20250624 14:19:48.363718 6073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:48.366371 6073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:48.367751 6073 hybrid_clock.cc:648] HybridClock initialized: now 1750774788367720 us; error 38 us; skew 500 ppm
I20250624 14:19:48.368551 6073 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:48.380664 6073 webserver.cc:469] Webserver started at http://127.2.14.62:34921/ using document root <none> and password file <none>
I20250624 14:19:48.381698 6073 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:48.381940 6073 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:48.390094 6073 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20250624 14:19:48.394647 6089 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:48.395756 6073 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250624 14:19:48.396078 6073 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7db2648ab6e441c38549ea70be0355e3"
format_stamp: "Formatted at 2025-06-24 14:19:18 on dist-test-slave-hs17"
I20250624 14:19:48.398038 6073 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:48.464824 6073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:48.466373 6073 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:48.466849 6073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:48.539247 6073 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.62:41603
I20250624 14:19:48.539330 6140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.62:41603 every 8 connection(s)
I20250624 14:19:48.542297 6073 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 14:19:48.551869 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 6073
I20250624 14:19:48.552526 6141 sys_catalog.cc:263] Verifying existing consensus state
I20250624 14:19:48.553398 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.1:36801
--local_ip_for_outbound_sockets=127.2.14.1
--tserver_master_addrs=127.2.14.62:41603
--webserver_port=41237
--webserver_interface=127.2.14.1
--builtin_ntp_servers=127.2.14.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:19:48.557410 6141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Bootstrap starting.
I20250624 14:19:48.567516 6141 log.cc:826] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:48.651244 6141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:48.652170 6141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Bootstrap complete.
I20250624 14:19:48.672883 6141 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:48.674950 6141 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db2648ab6e441c38549ea70be0355e3, State: Initialized, Role: FOLLOWER
I20250624 14:19:48.675832 6141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:48.676349 6141 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:19:48.676609 6141 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:19:48.676905 6141 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 5 FOLLOWER]: Advancing to term 6
I20250624 14:19:48.682404 6141 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:48.683122 6141 leader_election.cc:304] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7db2648ab6e441c38549ea70be0355e3; no voters:
I20250624 14:19:48.685432 6141 leader_election.cc:290] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250624 14:19:48.685987 6145 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 6 FOLLOWER]: Leader election won for term 6
I20250624 14:19:48.689705 6145 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [term 6 LEADER]: Becoming Leader. State: Replica: 7db2648ab6e441c38549ea70be0355e3, State: Running, Role: LEADER
I20250624 14:19:48.690696 6145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } }
I20250624 14:19:48.691097 6141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 14:19:48.702015 6146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "7db2648ab6e441c38549ea70be0355e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } } }
I20250624 14:19:48.702940 6146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: This master's current role is: LEADER
I20250624 14:19:48.705147 6147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7db2648ab6e441c38549ea70be0355e3. Latest consensus state: current_term: 6 leader_uuid: "7db2648ab6e441c38549ea70be0355e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db2648ab6e441c38549ea70be0355e3" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 41603 } } }
I20250624 14:19:48.705895 6147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3 [sys.catalog]: This master's current role is: LEADER
I20250624 14:19:48.717281 6151 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 14:19:48.731799 6151 catalog_manager.cc:671] Loaded metadata for table TestTable [id=14c2b97f6a6243469b67efd011f04f2b]
I20250624 14:19:48.734095 6151 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=76fe2eb3df674903a16acf89d22e43b5]
I20250624 14:19:48.736131 6151 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=d416107cfc6a4299acb39791b62918ca]
I20250624 14:19:48.746361 6151 tablet_loader.cc:96] loaded metadata for tablet 678437be284e4cf29a9dbb50aaa3727d (table TestTable1 [id=76fe2eb3df674903a16acf89d22e43b5])
I20250624 14:19:48.752647 6151 tablet_loader.cc:96] loaded metadata for tablet 8f2ce2bd8d124b8a858207ff7ba6e1db (table TestTable2 [id=d416107cfc6a4299acb39791b62918ca])
I20250624 14:19:48.754371 6151 tablet_loader.cc:96] loaded metadata for tablet df16f2e37e1e471db624a209f2ade936 (table TestTable [id=14c2b97f6a6243469b67efd011f04f2b])
I20250624 14:19:48.756093 6151 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 14:19:48.762684 6151 catalog_manager.cc:1261] Loaded cluster ID: db11613b64224d6a861fb54fb06c5787
I20250624 14:19:48.763070 6151 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 14:19:48.768868 6162 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250624 14:19:48.773203 6151 catalog_manager.cc:1506] Loading token signing keys...
I20250624 14:19:48.779632 6151 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 7db2648ab6e441c38549ea70be0355e3: Loaded TSK: 0
I20250624 14:19:48.781363 6151 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250624 14:19:48.917917 6143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:48.918449 6143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:48.918944 6143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:48.951056 6143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:48.951997 6143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.1
I20250624 14:19:48.987150 6143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.1:36801
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.14.1
--webserver_port=41237
--tserver_master_addrs=127.2.14.62:41603
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.1
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:48.988495 6143 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:48.990123 6143 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:49.007440 6169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:49.010964 6168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:49.010651 6171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:49.011036 6143 server_base.cc:1048] running on GCE node
I20250624 14:19:50.209455 6143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:50.212309 6143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:50.213825 6143 hybrid_clock.cc:648] HybridClock initialized: now 1750774790213772 us; error 56 us; skew 500 ppm
I20250624 14:19:50.214947 6143 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:50.223102 6143 webserver.cc:469] Webserver started at http://127.2.14.1:41237/ using document root <none> and password file <none>
I20250624 14:19:50.224433 6143 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:50.224757 6143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:50.235630 6143 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.003s
I20250624 14:19:50.241858 6178 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:50.243141 6143 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250624 14:19:50.243549 6143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "55a21f9c8e154fcc85e4b33617f4aec7"
format_stamp: "Formatted at 2025-06-24 14:19:20 on dist-test-slave-hs17"
I20250624 14:19:50.246454 6143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:50.332386 6143 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:50.334206 6143 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:50.334651 6143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:50.337786 6143 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:19:50.344293 6185 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250624 14:19:50.356022 6143 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250624 14:19:50.356248 6143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.003s sys 0.000s
I20250624 14:19:50.356556 6143 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250624 14:19:50.362004 6185 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: Bootstrap starting.
I20250624 14:19:50.366855 6143 ts_tablet_manager.cc:610] Registered 2 tablets
I20250624 14:19:50.367162 6143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.004s sys 0.004s
I20250624 14:19:50.441799 6185 log.cc:826] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:50.577845 6185 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:50.578967 6185 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: Bootstrap complete.
I20250624 14:19:50.580951 6185 ts_tablet_manager.cc:1397] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: Time spent bootstrapping tablet: real 0.219s user 0.163s sys 0.047s
I20250624 14:19:50.604732 6185 raft_consensus.cc:357] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:50.608219 6185 raft_consensus.cc:738] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a21f9c8e154fcc85e4b33617f4aec7, State: Initialized, Role: FOLLOWER
I20250624 14:19:50.609472 6185 consensus_queue.cc:260] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:50.619637 6185 ts_tablet_manager.cc:1428] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7: Time spent starting tablet: real 0.038s user 0.030s sys 0.004s
I20250624 14:19:50.620550 6185 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Bootstrap starting.
I20250624 14:19:50.630041 6143 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.1:36801
I20250624 14:19:50.630820 6293 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.1:36801 every 8 connection(s)
I20250624 14:19:50.634075 6143 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 14:19:50.636345 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 6143
I20250624 14:19:50.638111 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.2:39265
--local_ip_for_outbound_sockets=127.2.14.2
--tserver_master_addrs=127.2.14.62:41603
--webserver_port=45297
--webserver_interface=127.2.14.2
--builtin_ntp_servers=127.2.14.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:19:50.698896 6294 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41603
I20250624 14:19:50.699393 6294 heartbeater.cc:461] Registering TS with master...
I20250624 14:19:50.700685 6294 heartbeater.cc:507] Master 127.2.14.62:41603 requested a full tablet report, sending...
I20250624 14:19:50.706339 6106 ts_manager.cc:194] Registered new tserver with Master: 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1:36801)
I20250624 14:19:50.711525 6106 catalog_manager.cc:5582] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 reported cstate change: config changed from index -1 to 12, term changed from 0 to 2, VOTER 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2) added, VOTER 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1) added, VOTER 60919fec2a7d41739f65174c4e92014c (127.2.14.3) added. New cstate: current_term: 2 committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } } }
I20250624 14:19:50.791992 6106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.1:55527
I20250624 14:19:50.797828 6185 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:50.798568 6185 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Bootstrap complete.
I20250624 14:19:50.799813 6185 ts_tablet_manager.cc:1397] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Time spent bootstrapping tablet: real 0.179s user 0.140s sys 0.024s
I20250624 14:19:50.801396 6185 raft_consensus.cc:357] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:50.801962 6185 raft_consensus.cc:738] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a21f9c8e154fcc85e4b33617f4aec7, State: Initialized, Role: FOLLOWER
I20250624 14:19:50.802459 6185 consensus_queue.cc:260] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:50.803432 6294 heartbeater.cc:499] Master 127.2.14.62:41603 was elected leader, sending a full tablet report...
I20250624 14:19:50.804111 6185 ts_tablet_manager.cc:1428] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250624 14:19:51.030407 6295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:51.030895 6295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:51.031443 6295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:51.063899 6295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:51.064742 6295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.2
I20250624 14:19:51.101011 6295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.2:39265
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.14.2
--webserver_port=45297
--tserver_master_addrs=127.2.14.62:41603
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.2
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:51.102324 6295 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:51.104023 6295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:51.121938 6310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:52.166008 6316 raft_consensus.cc:491] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 14:19:52.166507 6316 raft_consensus.cc:513] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:52.168757 6316 leader_election.cc:290] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2:39265), 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843)
W20250624 14:19:52.172955 6182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.14.3:42843: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 14:19:52.176913 6180 leader_election.cc:336] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2:39265): Network error: Client connection negotiation failed: client connection to 127.2.14.2:39265: connect: Connection refused (error 111)
W20250624 14:19:52.177372 6182 leader_election.cc:336] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843): Network error: Client connection negotiation failed: client connection to 127.2.14.3:42843: connect: Connection refused (error 111)
I20250624 14:19:52.177816 6182 leader_election.cc:304] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55a21f9c8e154fcc85e4b33617f4aec7; no voters: 4112eeddb57742e48ba4b705d3a0c0a7, 60919fec2a7d41739f65174c4e92014c
I20250624 14:19:52.178639 6316 raft_consensus.cc:2747] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250624 14:19:52.261493 6316 raft_consensus.cc:491] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 14:19:52.262001 6316 raft_consensus.cc:513] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:52.263937 6316 leader_election.cc:290] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2:39265), 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843)
W20250624 14:19:52.267395 6180 leader_election.cc:336] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2:39265): Network error: Client connection negotiation failed: client connection to 127.2.14.2:39265: connect: Connection refused (error 111)
W20250624 14:19:52.268065 6182 leader_election.cc:336] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843): Network error: Client connection negotiation failed: client connection to 127.2.14.3:42843: connect: Connection refused (error 111)
I20250624 14:19:52.268523 6182 leader_election.cc:304] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55a21f9c8e154fcc85e4b33617f4aec7; no voters: 4112eeddb57742e48ba4b705d3a0c0a7, 60919fec2a7d41739f65174c4e92014c
I20250624 14:19:52.269304 6316 raft_consensus.cc:2747] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250624 14:19:51.122016 6309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:51.122767 6312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:52.304028 6311 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 14:19:52.304165 6295 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:19:52.308439 6295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:52.310649 6295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:52.311983 6295 hybrid_clock.cc:648] HybridClock initialized: now 1750774792311943 us; error 57 us; skew 500 ppm
I20250624 14:19:52.312754 6295 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:52.319644 6295 webserver.cc:469] Webserver started at http://127.2.14.2:45297/ using document root <none> and password file <none>
I20250624 14:19:52.320540 6295 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:52.320748 6295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:52.328487 6295 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250624 14:19:52.333187 6324 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:52.334185 6295 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250624 14:19:52.334466 6295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4112eeddb57742e48ba4b705d3a0c0a7"
format_stamp: "Formatted at 2025-06-24 14:19:22 on dist-test-slave-hs17"
I20250624 14:19:52.336419 6295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:52.389391 6295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:52.390956 6295 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:52.391438 6295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:52.394006 6295 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:19:52.399693 6331 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250624 14:19:52.411610 6295 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250624 14:19:52.411898 6295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.002s sys 0.000s
I20250624 14:19:52.412200 6295 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250624 14:19:52.417531 6331 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Bootstrap starting.
I20250624 14:19:52.420454 6295 ts_tablet_manager.cc:610] Registered 2 tablets
I20250624 14:19:52.420678 6295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.001s sys 0.007s
I20250624 14:19:52.484452 6331 log.cc:826] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:52.585069 6295 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.2:39265
I20250624 14:19:52.585242 6438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.2:39265 every 8 connection(s)
I20250624 14:19:52.588725 6295 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 14:19:52.596138 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 6295
I20250624 14:19:52.597973 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.3:42843
--local_ip_for_outbound_sockets=127.2.14.3
--tserver_master_addrs=127.2.14.62:41603
--webserver_port=34735
--webserver_interface=127.2.14.3
--builtin_ntp_servers=127.2.14.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250624 14:19:52.614805 6331 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:52.615769 6439 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41603
I20250624 14:19:52.615990 6331 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Bootstrap complete.
I20250624 14:19:52.616267 6439 heartbeater.cc:461] Registering TS with master...
I20250624 14:19:52.617549 6439 heartbeater.cc:507] Master 127.2.14.62:41603 requested a full tablet report, sending...
I20250624 14:19:52.617751 6331 ts_tablet_manager.cc:1397] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Time spent bootstrapping tablet: real 0.201s user 0.152s sys 0.044s
I20250624 14:19:52.623175 6106 ts_manager.cc:194] Registered new tserver with Master: 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2:39265)
I20250624 14:19:52.628401 6106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.2:57255
I20250624 14:19:52.635536 6331 raft_consensus.cc:357] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:52.638700 6331 raft_consensus.cc:738] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4112eeddb57742e48ba4b705d3a0c0a7, State: Initialized, Role: FOLLOWER
I20250624 14:19:52.639640 6331 consensus_queue.cc:260] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:52.642714 6439 heartbeater.cc:499] Master 127.2.14.62:41603 was elected leader, sending a full tablet report...
I20250624 14:19:52.643410 6331 ts_tablet_manager.cc:1428] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7: Time spent starting tablet: real 0.025s user 0.026s sys 0.000s
I20250624 14:19:52.644248 6331 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: Bootstrap starting.
I20250624 14:19:52.740710 6331 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:52.741408 6331 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: Bootstrap complete.
I20250624 14:19:52.742561 6331 ts_tablet_manager.cc:1397] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: Time spent bootstrapping tablet: real 0.098s user 0.087s sys 0.008s
I20250624 14:19:52.744175 6331 raft_consensus.cc:357] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:52.744628 6331 raft_consensus.cc:738] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4112eeddb57742e48ba4b705d3a0c0a7, State: Initialized, Role: FOLLOWER
I20250624 14:19:52.745137 6331 consensus_queue.cc:260] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:52.746632 6331 ts_tablet_manager.cc:1428] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250624 14:19:52.927047 6443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:19:52.927551 6443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:19:52.928113 6443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:19:52.960229 6443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:19:52.961097 6443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.3
I20250624 14:19:52.997193 6443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.3:42843
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.14.3
--webserver_port=34735
--tserver_master_addrs=127.2.14.62:41603
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.3
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:19:52.998546 6443 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:19:53.000314 6443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:19:53.017338 6451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:53.929081 6457 raft_consensus.cc:491] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 14:19:53.929570 6457 raft_consensus.cc:513] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:53.932089 6457 leader_election.cc:290] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1:36801), 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843)
W20250624 14:19:53.947803 6328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.14.3:42843: connect: Connection refused (error 111)
W20250624 14:19:53.951982 6328 leader_election.cc:336] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843): Network error: Client connection negotiation failed: client connection to 127.2.14.3:42843: connect: Connection refused (error 111)
I20250624 14:19:53.960398 6248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df16f2e37e1e471db624a209f2ade936" candidate_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" is_pre_election: true
I20250624 14:19:53.961622 6248 raft_consensus.cc:2466] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4112eeddb57742e48ba4b705d3a0c0a7 in term 2.
I20250624 14:19:53.963757 6326 leader_election.cc:304] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4112eeddb57742e48ba4b705d3a0c0a7, 55a21f9c8e154fcc85e4b33617f4aec7; no voters: 60919fec2a7d41739f65174c4e92014c
I20250624 14:19:53.964480 6457 raft_consensus.cc:2802] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250624 14:19:53.964829 6457 raft_consensus.cc:491] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 14:19:53.965091 6457 raft_consensus.cc:3058] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Advancing to term 3
I20250624 14:19:53.971380 6457 raft_consensus.cc:513] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:53.973027 6457 leader_election.cc:290] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 3 election: Requested vote from peers 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1:36801), 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843)
I20250624 14:19:53.973882 6248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df16f2e37e1e471db624a209f2ade936" candidate_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "55a21f9c8e154fcc85e4b33617f4aec7"
I20250624 14:19:53.974524 6248 raft_consensus.cc:3058] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Advancing to term 3
W20250624 14:19:53.977815 6328 leader_election.cc:336] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843): Network error: Client connection negotiation failed: client connection to 127.2.14.3:42843: connect: Connection refused (error 111)
I20250624 14:19:53.981679 6248 raft_consensus.cc:2466] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4112eeddb57742e48ba4b705d3a0c0a7 in term 3.
I20250624 14:19:53.982611 6326 leader_election.cc:304] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4112eeddb57742e48ba4b705d3a0c0a7, 55a21f9c8e154fcc85e4b33617f4aec7; no voters: 60919fec2a7d41739f65174c4e92014c
I20250624 14:19:53.983356 6457 raft_consensus.cc:2802] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 3 FOLLOWER]: Leader election won for term 3
I20250624 14:19:53.985965 6457 raft_consensus.cc:695] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 3 LEADER]: Becoming Leader. State: Replica: 4112eeddb57742e48ba4b705d3a0c0a7, State: Running, Role: LEADER
I20250624 14:19:53.987423 6457 consensus_queue.cc:237] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:54.002177 6106 catalog_manager.cc:5582] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 reported cstate change: term changed from 2 to 3, leader changed from <none> to 4112eeddb57742e48ba4b705d3a0c0a7 (127.2.14.2). New cstate: current_term: 3 leader_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250624 14:19:53.018280 6454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:19:53.017401 6452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:19:53.019651 6443 server_base.cc:1048] running on GCE node
I20250624 14:19:54.203526 6443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:19:54.205888 6443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:19:54.207240 6443 hybrid_clock.cc:648] HybridClock initialized: now 1750774794207196 us; error 55 us; skew 500 ppm
I20250624 14:19:54.208101 6443 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:19:54.219591 6443 webserver.cc:469] Webserver started at http://127.2.14.3:34735/ using document root <none> and password file <none>
I20250624 14:19:54.220592 6443 fs_manager.cc:362] Metadata directory not provided
I20250624 14:19:54.220829 6443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:19:54.228847 6443 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250624 14:19:54.233639 6472 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:19:54.234649 6443 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250624 14:19:54.234980 6443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "60919fec2a7d41739f65174c4e92014c"
format_stamp: "Formatted at 2025-06-24 14:19:24 on dist-test-slave-hs17"
I20250624 14:19:54.236984 6443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:19:54.292560 6443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:19:54.294116 6443 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:19:54.294564 6443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:19:54.297219 6443 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:19:54.303241 6479 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250624 14:19:54.319422 6443 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250624 14:19:54.319741 6443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.002s sys 0.001s
I20250624 14:19:54.320051 6443 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250624 14:19:54.325670 6479 tablet_bootstrap.cc:492] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Bootstrap starting.
I20250624 14:19:54.331485 6443 ts_tablet_manager.cc:610] Registered 3 tablets
I20250624 14:19:54.331750 6443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.012s sys 0.000s
I20250624 14:19:54.379452 6479 log.cc:826] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Log is configured to *not* fsync() on all Append() calls
I20250624 14:19:54.443610 6457 raft_consensus.cc:491] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 14:19:54.444172 6457 raft_consensus.cc:513] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:54.446450 6457 leader_election.cc:290] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1:36801), 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843)
I20250624 14:19:54.447345 6248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "678437be284e4cf29a9dbb50aaa3727d" candidate_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" is_pre_election: true
I20250624 14:19:54.448025 6248 raft_consensus.cc:2466] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4112eeddb57742e48ba4b705d3a0c0a7 in term 2.
I20250624 14:19:54.449154 6326 leader_election.cc:304] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4112eeddb57742e48ba4b705d3a0c0a7, 55a21f9c8e154fcc85e4b33617f4aec7; no voters:
I20250624 14:19:54.450186 6457 raft_consensus.cc:2802] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250624 14:19:54.450596 6457 raft_consensus.cc:491] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 14:19:54.451026 6457 raft_consensus.cc:3058] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 2 FOLLOWER]: Advancing to term 3
W20250624 14:19:54.452399 6328 leader_election.cc:336] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843): Network error: Client connection negotiation failed: client connection to 127.2.14.3:42843: connect: Connection refused (error 111)
W20250624 14:19:54.455471 6328 consensus_peers.cc:487] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 -> Peer 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843): Couldn't send request to peer 60919fec2a7d41739f65174c4e92014c. Status: Network error: Client connection negotiation failed: client connection to 127.2.14.3:42843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250624 14:19:54.459785 6457 raft_consensus.cc:513] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:54.462281 6457 leader_election.cc:290] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 3 election: Requested vote from peers 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1:36801), 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843)
I20250624 14:19:54.463889 6248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "678437be284e4cf29a9dbb50aaa3727d" candidate_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "55a21f9c8e154fcc85e4b33617f4aec7"
I20250624 14:19:54.464445 6248 raft_consensus.cc:3058] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 2 FOLLOWER]: Advancing to term 3
I20250624 14:19:54.471047 6248 raft_consensus.cc:2466] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4112eeddb57742e48ba4b705d3a0c0a7 in term 3.
W20250624 14:19:54.470986 6328 leader_election.cc:336] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843): Network error: Client connection negotiation failed: client connection to 127.2.14.3:42843: connect: Connection refused (error 111)
I20250624 14:19:54.472231 6326 leader_election.cc:304] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4112eeddb57742e48ba4b705d3a0c0a7, 55a21f9c8e154fcc85e4b33617f4aec7; no voters: 60919fec2a7d41739f65174c4e92014c
I20250624 14:19:54.473218 6457 raft_consensus.cc:2802] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 3 FOLLOWER]: Leader election won for term 3
I20250624 14:19:54.473639 6457 raft_consensus.cc:695] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [term 3 LEADER]: Becoming Leader. State: Replica: 4112eeddb57742e48ba4b705d3a0c0a7, State: Running, Role: LEADER
I20250624 14:19:54.474409 6457 consensus_queue.cc:237] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:54.482721 6106 catalog_manager.cc:5582] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250624 14:19:54.485031 6248 raft_consensus.cc:1273] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 3 FOLLOWER]: Refusing update from remote peer 4112eeddb57742e48ba4b705d3a0c0a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250624 14:19:54.487177 6457 consensus_queue.cc:1035] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250624 14:19:54.489388 6479 tablet_bootstrap.cc:492] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:54.491611 6479 tablet_bootstrap.cc:492] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Bootstrap complete.
I20250624 14:19:54.493654 6479 ts_tablet_manager.cc:1397] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Time spent bootstrapping tablet: real 0.168s user 0.122s sys 0.038s
I20250624 14:19:54.519270 6479 raft_consensus.cc:357] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } }
I20250624 14:19:54.522629 6479 raft_consensus.cc:738] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60919fec2a7d41739f65174c4e92014c, State: Initialized, Role: FOLLOWER
I20250624 14:19:54.523578 6479 consensus_queue.cc:260] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } }
I20250624 14:19:54.524156 6479 raft_consensus.cc:397] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:19:54.524371 6479 raft_consensus.cc:491] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:19:54.524698 6479 raft_consensus.cc:3058] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Advancing to term 3
I20250624 14:19:54.531214 6479 raft_consensus.cc:513] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } }
I20250624 14:19:54.532131 6479 leader_election.cc:304] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60919fec2a7d41739f65174c4e92014c; no voters:
I20250624 14:19:54.535853 6479 leader_election.cc:290] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [CANDIDATE]: Term 3 election: Requested vote from peers
I20250624 14:19:54.536193 6571 raft_consensus.cc:2802] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 3 FOLLOWER]: Leader election won for term 3
I20250624 14:19:54.547224 6571 raft_consensus.cc:695] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [term 3 LEADER]: Becoming Leader. State: Replica: 60919fec2a7d41739f65174c4e92014c, State: Running, Role: LEADER
I20250624 14:19:54.548604 6571 consensus_queue.cc:237] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } }
I20250624 14:19:54.558125 6479 ts_tablet_manager.cc:1428] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c: Time spent starting tablet: real 0.064s user 0.040s sys 0.018s
I20250624 14:19:54.559047 6479 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: Bootstrap starting.
I20250624 14:19:54.570109 6394 consensus_queue.cc:237] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } }
I20250624 14:19:54.578055 6248 raft_consensus.cc:1273] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 3 FOLLOWER]: Refusing update from remote peer 4112eeddb57742e48ba4b705d3a0c0a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250624 14:19:54.579972 6457 consensus_queue.cc:1035] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250624 14:19:54.588480 6457 raft_consensus.cc:2953] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 60919fec2a7d41739f65174c4e92014c (127.2.14.3) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } }
I20250624 14:19:54.590467 6248 raft_consensus.cc:2953] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 60919fec2a7d41739f65174c4e92014c (127.2.14.3) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } }
I20250624 14:19:54.624009 6091 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet df16f2e37e1e471db624a209f2ade936 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250624 14:19:54.631095 6106 catalog_manager.cc:5582] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 reported cstate change: config changed from index 12 to 14, VOTER 60919fec2a7d41739f65174c4e92014c (127.2.14.3) evicted. New cstate: current_term: 3 leader_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250624 14:19:54.645006 6106 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet df16f2e37e1e471db624a209f2ade936 on TS 60919fec2a7d41739f65174c4e92014c: Not found: failed to reset TS proxy: Could not find TS for UUID 60919fec2a7d41739f65174c4e92014c
I20250624 14:19:54.648115 6394 consensus_queue.cc:237] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 12, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } }
I20250624 14:19:54.650924 6462 raft_consensus.cc:2953] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } }
I20250624 14:19:54.661067 6091 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet df16f2e37e1e471db624a209f2ade936 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250624 14:19:54.663913 6106 catalog_manager.cc:5582] T df16f2e37e1e471db624a209f2ade936 P 4112eeddb57742e48ba4b705d3a0c0a7 reported cstate change: config changed from index 14 to 15, VOTER 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1) evicted. New cstate: current_term: 3 leader_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250624 14:19:54.696610 6228 tablet_service.cc:1515] Processing DeleteTablet for tablet df16f2e37e1e471db624a209f2ade936 with delete_type TABLET_DATA_TOMBSTONED (TS 55a21f9c8e154fcc85e4b33617f4aec7 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:34088
W20250624 14:19:54.697989 6091 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet df16f2e37e1e471db624a209f2ade936 on TS 60919fec2a7d41739f65174c4e92014c failed: Not found: failed to reset TS proxy: Could not find TS for UUID 60919fec2a7d41739f65174c4e92014c
I20250624 14:19:54.700255 6592 tablet_replica.cc:331] stopping tablet replica
I20250624 14:19:54.700870 6592 raft_consensus.cc:2241] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250624 14:19:54.701375 6592 raft_consensus.cc:2270] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250624 14:19:54.705855 6592 ts_tablet_manager.cc:1905] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250624 14:19:54.715395 6443 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.3:42843
I20250624 14:19:54.715740 6601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.3:42843 every 8 connection(s)
I20250624 14:19:54.719017 6443 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 14:19:54.720984 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 6443
I20250624 14:19:54.726003 6479 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:54.727336 6479 tablet_bootstrap.cc:492] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: Bootstrap complete.
I20250624 14:19:54.729566 6479 ts_tablet_manager.cc:1397] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: Time spent bootstrapping tablet: real 0.171s user 0.140s sys 0.020s
I20250624 14:19:54.722641 6592 ts_tablet_manager.cc:1918] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20250624 14:19:54.734105 6592 log.cc:1199] T df16f2e37e1e471db624a209f2ade936 P 55a21f9c8e154fcc85e4b33617f4aec7: Deleting WAL directory at /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/wals/df16f2e37e1e471db624a209f2ade936
I20250624 14:19:54.737758 6479 raft_consensus.cc:357] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:54.743333 6479 raft_consensus.cc:738] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60919fec2a7d41739f65174c4e92014c, State: Initialized, Role: FOLLOWER
I20250624 14:19:54.744242 6479 consensus_queue.cc:260] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } } peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:54.746225 6479 ts_tablet_manager.cc:1428] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250624 14:19:54.746704 6091 catalog_manager.cc:4928] TS 55a21f9c8e154fcc85e4b33617f4aec7 (127.2.14.1:36801): tablet df16f2e37e1e471db624a209f2ade936 (table TestTable [id=14c2b97f6a6243469b67efd011f04f2b]) successfully deleted
I20250624 14:19:54.746994 6479 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: Bootstrap starting.
I20250624 14:19:54.759369 6602 heartbeater.cc:344] Connected to a master server at 127.2.14.62:41603
I20250624 14:19:54.759862 6602 heartbeater.cc:461] Registering TS with master...
I20250624 14:19:54.760872 6602 heartbeater.cc:507] Master 127.2.14.62:41603 requested a full tablet report, sending...
I20250624 14:19:54.765739 6106 ts_manager.cc:194] Registered new tserver with Master: 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843)
I20250624 14:19:54.768767 6106 catalog_manager.cc:5582] T 8f2ce2bd8d124b8a858207ff7ba6e1db P 60919fec2a7d41739f65174c4e92014c reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "60919fec2a7d41739f65174c4e92014c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } health_report { overall_health: HEALTHY } } }
I20250624 14:19:54.773380 2104 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 14:19:54.778718 2104 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 14:19:54.780259 6106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.3:38573
W20250624 14:19:54.782068 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250624 14:19:54.784173 6602 heartbeater.cc:499] Master 127.2.14.62:41603 was elected leader, sending a full tablet report...
I20250624 14:19:54.854061 6479 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:19:54.854776 6479 tablet_bootstrap.cc:492] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: Bootstrap complete.
I20250624 14:19:54.856001 6479 ts_tablet_manager.cc:1397] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: Time spent bootstrapping tablet: real 0.109s user 0.093s sys 0.015s
I20250624 14:19:54.858088 6479 raft_consensus.cc:357] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:54.858548 6479 raft_consensus.cc:738] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60919fec2a7d41739f65174c4e92014c, State: Initialized, Role: FOLLOWER
I20250624 14:19:54.859055 6479 consensus_queue.cc:260] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } } peers { permanent_uuid: "4112eeddb57742e48ba4b705d3a0c0a7" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false } }
I20250624 14:19:54.860541 6479 ts_tablet_manager.cc:1428] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250624 14:19:54.868729 6522 tablet_service.cc:1515] Processing DeleteTablet for tablet df16f2e37e1e471db624a209f2ade936 with delete_type TABLET_DATA_TOMBSTONED (TS 60919fec2a7d41739f65174c4e92014c not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:42140
I20250624 14:19:54.873798 6610 tablet_replica.cc:331] stopping tablet replica
I20250624 14:19:54.874451 6610 raft_consensus.cc:2241] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Raft consensus shutting down.
I20250624 14:19:54.874886 6610 raft_consensus.cc:2270] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250624 14:19:54.877815 6610 ts_tablet_manager.cc:1905] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250624 14:19:54.888393 6610 ts_tablet_manager.cc:1918] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250624 14:19:54.888715 6610 log.cc:1199] T df16f2e37e1e471db624a209f2ade936 P 60919fec2a7d41739f65174c4e92014c: Deleting WAL directory at /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal/wals/df16f2e37e1e471db624a209f2ade936
I20250624 14:19:54.890453 6093 catalog_manager.cc:4928] TS 60919fec2a7d41739f65174c4e92014c (127.2.14.3:42843): tablet df16f2e37e1e471db624a209f2ade936 (table TestTable [id=14c2b97f6a6243469b67efd011f04f2b]) successfully deleted
I20250624 14:19:54.923442 6248 raft_consensus.cc:1273] T 678437be284e4cf29a9dbb50aaa3727d P 55a21f9c8e154fcc85e4b33617f4aec7 [term 3 FOLLOWER]: Refusing update from remote peer 4112eeddb57742e48ba4b705d3a0c0a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250624 14:19:54.924568 6462 consensus_queue.cc:1035] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a21f9c8e154fcc85e4b33617f4aec7" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 36801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250624 14:19:54.942451 6542 raft_consensus.cc:3058] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c [term 2 FOLLOWER]: Advancing to term 3
I20250624 14:19:54.947160 6542 raft_consensus.cc:1273] T 678437be284e4cf29a9dbb50aaa3727d P 60919fec2a7d41739f65174c4e92014c [term 3 FOLLOWER]: Refusing update from remote peer 4112eeddb57742e48ba4b705d3a0c0a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250624 14:19:54.948665 6612 consensus_queue.cc:1035] T 678437be284e4cf29a9dbb50aaa3727d P 4112eeddb57742e48ba4b705d3a0c0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60919fec2a7d41739f65174c4e92014c" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 42843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20250624 14:19:55.785883 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:19:56.789556 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:19:57.793429 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:19:58.796888 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:19:59.800282 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:20:00.803740 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:20:01.807112 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:20:02.811403 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:20:03.815222 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:20:04.818567 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:20:05.822007 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:20:06.825721 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:20:07.829900 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:20:08.833519 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:20:09.836900 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:20:10.840443 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:20:11.843940 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:20:12.848165 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250624 14:20:13.851828 2104 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet df16f2e37e1e471db624a209f2ade936: tablet_id: "df16f2e37e1e471db624a209f2ade936" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250624 14:20:14.855396 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 6143
W20250624 14:20:14.883546 6326 connection.cc:537] client connection to 127.2.14.1:36801 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250624 14:20:14.884080 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 6295
I20250624 14:20:14.916433 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 6443
I20250624 14:20:14.943449 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 6073
2025-06-24T14:20:14Z chronyd exiting
I20250624 14:20:14.995847 2104 test_util.cc:183] -----------------------------------------------
I20250624 14:20:14.996055 2104 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750774691568470-2104-0
[ FAILED ] AdminCliTest.TestRebuildTables (58253 ms)
[----------] 5 tests from AdminCliTest (123357 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250624 14:20:14.999991 2104 test_util.cc:276] Using random seed: 116262300
I20250624 14:20:15.004333 2104 ts_itest-base.cc:115] Starting cluster with:
I20250624 14:20:15.004482 2104 ts_itest-base.cc:116] --------------
I20250624 14:20:15.004637 2104 ts_itest-base.cc:117] 5 tablet servers
I20250624 14:20:15.004801 2104 ts_itest-base.cc:118] 3 replicas per TS
I20250624 14:20:15.004941 2104 ts_itest-base.cc:119] --------------
2025-06-24T14:20:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T14:20:15Z Disabled control of system clock
I20250624 14:20:15.052701 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:38655
--webserver_interface=127.2.14.62
--webserver_port=0
--builtin_ntp_servers=127.2.14.20:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.14.62:38655
--raft_prepare_replacement_before_eviction=true with env {}
W20250624 14:20:15.360543 6641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:20:15.361152 6641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:20:15.361613 6641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:20:15.393384 6641 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 14:20:15.393802 6641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 14:20:15.394055 6641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:20:15.394292 6641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 14:20:15.394521 6641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 14:20:15.430963 6641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.14.62:38655
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:38655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:20:15.432324 6641 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:20:15.434006 6641 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:20:15.449331 6650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:15.449390 6648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:15.450444 6641 server_base.cc:1048] running on GCE node
W20250624 14:20:15.450145 6647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:16.635624 6641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:20:16.638350 6641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:20:16.639739 6641 hybrid_clock.cc:648] HybridClock initialized: now 1750774816639693 us; error 52 us; skew 500 ppm
I20250624 14:20:16.640535 6641 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:20:16.651628 6641 webserver.cc:469] Webserver started at http://127.2.14.62:37237/ using document root <none> and password file <none>
I20250624 14:20:16.652735 6641 fs_manager.cc:362] Metadata directory not provided
I20250624 14:20:16.652949 6641 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:20:16.653386 6641 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:20:16.658265 6641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d0a1ac85b1644855afdf266f726569c0"
format_stamp: "Formatted at 2025-06-24 14:20:16 on dist-test-slave-hs17"
I20250624 14:20:16.659399 6641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d0a1ac85b1644855afdf266f726569c0"
format_stamp: "Formatted at 2025-06-24 14:20:16 on dist-test-slave-hs17"
I20250624 14:20:16.667166 6641 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250624 14:20:16.672915 6657 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:16.674000 6641 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250624 14:20:16.674333 6641 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d0a1ac85b1644855afdf266f726569c0"
format_stamp: "Formatted at 2025-06-24 14:20:16 on dist-test-slave-hs17"
I20250624 14:20:16.674665 6641 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:20:16.728793 6641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:20:16.730325 6641 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:20:16.730794 6641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:20:16.801760 6641 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.62:38655
I20250624 14:20:16.801822 6708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.62:38655 every 8 connection(s)
I20250624 14:20:16.804515 6641 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 14:20:16.809633 6709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:20:16.811451 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 6641
I20250624 14:20:16.811926 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 14:20:16.830627 6709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0: Bootstrap starting.
I20250624 14:20:16.837118 6709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0: Neither blocks nor log segments found. Creating new log.
I20250624 14:20:16.838790 6709 log.cc:826] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0: Log is configured to *not* fsync() on all Append() calls
I20250624 14:20:16.843407 6709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0: No bootstrap required, opened a new log
I20250624 14:20:16.862030 6709 raft_consensus.cc:357] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a1ac85b1644855afdf266f726569c0" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 38655 } }
I20250624 14:20:16.862821 6709 raft_consensus.cc:383] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:20:16.863131 6709 raft_consensus.cc:738] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a1ac85b1644855afdf266f726569c0, State: Initialized, Role: FOLLOWER
I20250624 14:20:16.864096 6709 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a1ac85b1644855afdf266f726569c0" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 38655 } }
I20250624 14:20:16.864621 6709 raft_consensus.cc:397] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:20:16.864920 6709 raft_consensus.cc:491] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:20:16.865219 6709 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:20:16.869527 6709 raft_consensus.cc:513] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a1ac85b1644855afdf266f726569c0" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 38655 } }
I20250624 14:20:16.870237 6709 leader_election.cc:304] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0a1ac85b1644855afdf266f726569c0; no voters:
I20250624 14:20:16.872118 6709 leader_election.cc:290] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 14:20:16.872887 6714 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:20:16.875133 6714 raft_consensus.cc:695] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [term 1 LEADER]: Becoming Leader. State: Replica: d0a1ac85b1644855afdf266f726569c0, State: Running, Role: LEADER
I20250624 14:20:16.875854 6714 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a1ac85b1644855afdf266f726569c0" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 38655 } }
I20250624 14:20:16.876394 6709 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 14:20:16.885879 6716 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0a1ac85b1644855afdf266f726569c0. Latest consensus state: current_term: 1 leader_uuid: "d0a1ac85b1644855afdf266f726569c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a1ac85b1644855afdf266f726569c0" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 38655 } } }
I20250624 14:20:16.886538 6716 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [sys.catalog]: This master's current role is: LEADER
I20250624 14:20:16.887393 6715 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0a1ac85b1644855afdf266f726569c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a1ac85b1644855afdf266f726569c0" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 38655 } } }
I20250624 14:20:16.888154 6715 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0 [sys.catalog]: This master's current role is: LEADER
I20250624 14:20:16.892864 6722 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 14:20:16.903985 6722 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 14:20:16.920172 6722 catalog_manager.cc:1349] Generated new cluster ID: c049f78a254948da8bd6badc5c2b3f42
I20250624 14:20:16.920492 6722 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 14:20:16.937722 6722 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 14:20:16.939219 6722 catalog_manager.cc:1506] Loading token signing keys...
I20250624 14:20:16.958210 6722 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d0a1ac85b1644855afdf266f726569c0: Generated new TSK 0
I20250624 14:20:16.959321 6722 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 14:20:16.981976 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.1:0
--local_ip_for_outbound_sockets=127.2.14.1
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:38655
--builtin_ntp_servers=127.2.14.20:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250624 14:20:17.298054 6733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:20:17.298585 6733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:20:17.299086 6733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:20:17.330961 6733 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 14:20:17.331393 6733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:20:17.332223 6733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.1
I20250624 14:20:17.368130 6733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:38655
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.1
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:20:17.369503 6733 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:20:17.371155 6733 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:20:17.389384 6740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:17.391886 6739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:17.391068 6742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:17.391180 6733 server_base.cc:1048] running on GCE node
I20250624 14:20:18.583426 6733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:20:18.586277 6733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:20:18.587766 6733 hybrid_clock.cc:648] HybridClock initialized: now 1750774818587716 us; error 50 us; skew 500 ppm
I20250624 14:20:18.588857 6733 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:20:18.597126 6733 webserver.cc:469] Webserver started at http://127.2.14.1:35165/ using document root <none> and password file <none>
I20250624 14:20:18.598107 6733 fs_manager.cc:362] Metadata directory not provided
I20250624 14:20:18.598338 6733 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:20:18.598805 6733 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:20:18.603487 6733 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "93b1ddd3032141b8a99e9f6df495796e"
format_stamp: "Formatted at 2025-06-24 14:20:18 on dist-test-slave-hs17"
I20250624 14:20:18.604658 6733 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "93b1ddd3032141b8a99e9f6df495796e"
format_stamp: "Formatted at 2025-06-24 14:20:18 on dist-test-slave-hs17"
I20250624 14:20:18.612362 6733 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250624 14:20:18.618487 6749 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:18.619683 6733 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.001s
I20250624 14:20:18.620000 6733 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "93b1ddd3032141b8a99e9f6df495796e"
format_stamp: "Formatted at 2025-06-24 14:20:18 on dist-test-slave-hs17"
I20250624 14:20:18.620312 6733 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:20:18.677580 6733 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:20:18.679082 6733 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:20:18.679518 6733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:20:18.682140 6733 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:20:18.686450 6733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:20:18.686673 6733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:18.686971 6733 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:20:18.687132 6733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:18.823505 6733 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.1:35585
I20250624 14:20:18.823688 6861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.1:35585 every 8 connection(s)
I20250624 14:20:18.826097 6733 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 14:20:18.828029 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 6733
I20250624 14:20:18.828457 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 14:20:18.835536 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.2:0
--local_ip_for_outbound_sockets=127.2.14.2
--webserver_interface=127.2.14.2
--webserver_port=0
--tserver_master_addrs=127.2.14.62:38655
--builtin_ntp_servers=127.2.14.20:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250624 14:20:18.849524 6862 heartbeater.cc:344] Connected to a master server at 127.2.14.62:38655
I20250624 14:20:18.849969 6862 heartbeater.cc:461] Registering TS with master...
I20250624 14:20:18.850996 6862 heartbeater.cc:507] Master 127.2.14.62:38655 requested a full tablet report, sending...
I20250624 14:20:18.853538 6674 ts_manager.cc:194] Registered new tserver with Master: 93b1ddd3032141b8a99e9f6df495796e (127.2.14.1:35585)
I20250624 14:20:18.856123 6674 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.1:52937
W20250624 14:20:19.144886 6866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:20:19.145443 6866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:20:19.145907 6866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:20:19.176944 6866 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 14:20:19.177306 6866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:20:19.178131 6866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.2
I20250624 14:20:19.213459 6866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.14.2
--webserver_port=0
--tserver_master_addrs=127.2.14.62:38655
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.2
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:20:19.214764 6866 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:20:19.216434 6866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:20:19.234203 6873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:19.859913 6862 heartbeater.cc:499] Master 127.2.14.62:38655 was elected leader, sending a full tablet report...
W20250624 14:20:19.234673 6872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:19.234390 6875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:20.407721 6874 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 14:20:20.407847 6866 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:20:20.412065 6866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:20:20.414819 6866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:20:20.416265 6866 hybrid_clock.cc:648] HybridClock initialized: now 1750774820416222 us; error 56 us; skew 500 ppm
I20250624 14:20:20.417105 6866 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:20:20.424003 6866 webserver.cc:469] Webserver started at http://127.2.14.2:33065/ using document root <none> and password file <none>
I20250624 14:20:20.424965 6866 fs_manager.cc:362] Metadata directory not provided
I20250624 14:20:20.425166 6866 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:20:20.425642 6866 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:20:20.430171 6866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "26fa16017d034c6792c347dc57d345ba"
format_stamp: "Formatted at 2025-06-24 14:20:20 on dist-test-slave-hs17"
I20250624 14:20:20.431296 6866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "26fa16017d034c6792c347dc57d345ba"
format_stamp: "Formatted at 2025-06-24 14:20:20 on dist-test-slave-hs17"
I20250624 14:20:20.438505 6866 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250624 14:20:20.444226 6882 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:20.445295 6866 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250624 14:20:20.445637 6866 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "26fa16017d034c6792c347dc57d345ba"
format_stamp: "Formatted at 2025-06-24 14:20:20 on dist-test-slave-hs17"
I20250624 14:20:20.445961 6866 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:20:20.493422 6866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:20:20.494947 6866 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:20:20.495373 6866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:20:20.498486 6866 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:20:20.502812 6866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:20:20.503041 6866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:20.503296 6866 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:20:20.503456 6866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:20.641115 6866 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.2:33661
I20250624 14:20:20.641227 6994 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.2:33661 every 8 connection(s)
I20250624 14:20:20.643826 6866 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250624 14:20:20.650074 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 6866
I20250624 14:20:20.650635 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250624 14:20:20.658299 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.3:0
--local_ip_for_outbound_sockets=127.2.14.3
--webserver_interface=127.2.14.3
--webserver_port=0
--tserver_master_addrs=127.2.14.62:38655
--builtin_ntp_servers=127.2.14.20:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250624 14:20:20.672703 6995 heartbeater.cc:344] Connected to a master server at 127.2.14.62:38655
I20250624 14:20:20.673204 6995 heartbeater.cc:461] Registering TS with master...
I20250624 14:20:20.674985 6995 heartbeater.cc:507] Master 127.2.14.62:38655 requested a full tablet report, sending...
I20250624 14:20:20.677809 6674 ts_manager.cc:194] Registered new tserver with Master: 26fa16017d034c6792c347dc57d345ba (127.2.14.2:33661)
I20250624 14:20:20.679842 6674 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.2:37467
W20250624 14:20:20.991863 6999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:20:20.992338 6999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:20:20.992792 6999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:20:21.024724 6999 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 14:20:21.025099 6999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:20:21.025828 6999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.3
I20250624 14:20:21.060606 6999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.14.3
--webserver_port=0
--tserver_master_addrs=127.2.14.62:38655
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.3
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:20:21.061931 6999 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:20:21.063552 6999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:20:21.081306 7005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:21.683713 6995 heartbeater.cc:499] Master 127.2.14.62:38655 was elected leader, sending a full tablet report...
W20250624 14:20:21.081400 7008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:21.083581 6999 server_base.cc:1048] running on GCE node
W20250624 14:20:21.081408 7006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:22.245654 6999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:20:22.247993 6999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:20:22.249370 6999 hybrid_clock.cc:648] HybridClock initialized: now 1750774822249342 us; error 47 us; skew 500 ppm
I20250624 14:20:22.250191 6999 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:20:22.260243 6999 webserver.cc:469] Webserver started at http://127.2.14.3:41915/ using document root <none> and password file <none>
I20250624 14:20:22.261229 6999 fs_manager.cc:362] Metadata directory not provided
I20250624 14:20:22.261444 6999 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:20:22.261925 6999 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:20:22.266474 6999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9016a354df354ae9a10f8cb6ac60d0be"
format_stamp: "Formatted at 2025-06-24 14:20:22 on dist-test-slave-hs17"
I20250624 14:20:22.267616 6999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9016a354df354ae9a10f8cb6ac60d0be"
format_stamp: "Formatted at 2025-06-24 14:20:22 on dist-test-slave-hs17"
I20250624 14:20:22.274787 6999 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20250624 14:20:22.280562 7016 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:22.281682 6999 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250624 14:20:22.282008 6999 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9016a354df354ae9a10f8cb6ac60d0be"
format_stamp: "Formatted at 2025-06-24 14:20:22 on dist-test-slave-hs17"
I20250624 14:20:22.282364 6999 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:20:22.330273 6999 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:20:22.331733 6999 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:20:22.332144 6999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:20:22.334605 6999 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:20:22.338841 6999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:20:22.339049 6999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:22.339260 6999 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:20:22.339391 6999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:22.476537 6999 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.3:41413
I20250624 14:20:22.476650 7128 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.3:41413 every 8 connection(s)
I20250624 14:20:22.479147 6999 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250624 14:20:22.484500 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 6999
I20250624 14:20:22.485293 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250624 14:20:22.492501 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.4:0
--local_ip_for_outbound_sockets=127.2.14.4
--webserver_interface=127.2.14.4
--webserver_port=0
--tserver_master_addrs=127.2.14.62:38655
--builtin_ntp_servers=127.2.14.20:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250624 14:20:22.501081 7129 heartbeater.cc:344] Connected to a master server at 127.2.14.62:38655
I20250624 14:20:22.501534 7129 heartbeater.cc:461] Registering TS with master...
I20250624 14:20:22.502784 7129 heartbeater.cc:507] Master 127.2.14.62:38655 requested a full tablet report, sending...
I20250624 14:20:22.505347 6674 ts_manager.cc:194] Registered new tserver with Master: 9016a354df354ae9a10f8cb6ac60d0be (127.2.14.3:41413)
I20250624 14:20:22.507174 6674 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.3:53831
W20250624 14:20:22.817385 7133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:20:22.817916 7133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:20:22.818413 7133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:20:22.857039 7133 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 14:20:22.857463 7133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:20:22.858264 7133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.4
I20250624 14:20:22.896364 7133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.14.4
--webserver_port=0
--tserver_master_addrs=127.2.14.62:38655
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.4
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:20:22.897737 7133 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:20:22.899415 7133 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:20:22.917212 7140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:23.510982 7129 heartbeater.cc:499] Master 127.2.14.62:38655 was elected leader, sending a full tablet report...
W20250624 14:20:22.917245 7139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:22.919003 7142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:24.098706 7141 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 14:20:24.098839 7133 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:20:24.102866 7133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:20:24.105078 7133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:20:24.106426 7133 hybrid_clock.cc:648] HybridClock initialized: now 1750774824106390 us; error 58 us; skew 500 ppm
I20250624 14:20:24.107204 7133 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:20:24.113461 7133 webserver.cc:469] Webserver started at http://127.2.14.4:34413/ using document root <none> and password file <none>
I20250624 14:20:24.114368 7133 fs_manager.cc:362] Metadata directory not provided
I20250624 14:20:24.114559 7133 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:20:24.114974 7133 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:20:24.119863 7133 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1d8dc6b82dda4e029cdaf8b75c52e55f"
format_stamp: "Formatted at 2025-06-24 14:20:24 on dist-test-slave-hs17"
I20250624 14:20:24.120942 7133 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1d8dc6b82dda4e029cdaf8b75c52e55f"
format_stamp: "Formatted at 2025-06-24 14:20:24 on dist-test-slave-hs17"
I20250624 14:20:24.128057 7133 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250624 14:20:24.133708 7149 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:24.134737 7133 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250624 14:20:24.135059 7133 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1d8dc6b82dda4e029cdaf8b75c52e55f"
format_stamp: "Formatted at 2025-06-24 14:20:24 on dist-test-slave-hs17"
I20250624 14:20:24.135393 7133 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:20:24.184535 7133 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:20:24.186101 7133 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:20:24.186523 7133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:20:24.191722 7133 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:20:24.197413 7133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:20:24.197623 7133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:24.197889 7133 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:20:24.198045 7133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:24.332191 7133 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.4:42561
I20250624 14:20:24.332297 7261 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.4:42561 every 8 connection(s)
I20250624 14:20:24.334812 7133 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250624 14:20:24.335924 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 7133
I20250624 14:20:24.336455 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250624 14:20:24.344938 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.5:0
--local_ip_for_outbound_sockets=127.2.14.5
--webserver_interface=127.2.14.5
--webserver_port=0
--tserver_master_addrs=127.2.14.62:38655
--builtin_ntp_servers=127.2.14.20:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250624 14:20:24.358078 7262 heartbeater.cc:344] Connected to a master server at 127.2.14.62:38655
I20250624 14:20:24.358686 7262 heartbeater.cc:461] Registering TS with master...
I20250624 14:20:24.360186 7262 heartbeater.cc:507] Master 127.2.14.62:38655 requested a full tablet report, sending...
I20250624 14:20:24.362761 6674 ts_manager.cc:194] Registered new tserver with Master: 1d8dc6b82dda4e029cdaf8b75c52e55f (127.2.14.4:42561)
I20250624 14:20:24.364431 6674 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.4:43395
W20250624 14:20:24.538131 7125 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac060 after lost signal to thread 7000
W20250624 14:20:24.539054 7125 debug-util.cc:398] Leaking SignalData structure 0x7b08000acf40 after lost signal to thread 7128
W20250624 14:20:24.675736 7266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:20:24.676326 7266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:20:24.676863 7266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:20:24.712284 7266 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250624 14:20:24.712718 7266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:20:24.713505 7266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.5
I20250624 14:20:24.750887 7266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.14.5
--webserver_port=0
--tserver_master_addrs=127.2.14.62:38655
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.5
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:20:24.752377 7266 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:20:24.754336 7266 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:20:24.775811 7273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:25.370048 7262 heartbeater.cc:499] Master 127.2.14.62:38655 was elected leader, sending a full tablet report...
W20250624 14:20:24.777233 7272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:24.776233 7275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:26.287274 7274 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1509 milliseconds
I20250624 14:20:26.287364 7266 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:20:26.288652 7266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:20:26.291476 7266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:20:26.292951 7266 hybrid_clock.cc:648] HybridClock initialized: now 1750774826292889 us; error 83 us; skew 500 ppm
I20250624 14:20:26.293754 7266 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:20:26.301117 7266 webserver.cc:469] Webserver started at http://127.2.14.5:39049/ using document root <none> and password file <none>
I20250624 14:20:26.302107 7266 fs_manager.cc:362] Metadata directory not provided
I20250624 14:20:26.302309 7266 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:20:26.302778 7266 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:20:26.307637 7266 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a1845000ee254a6b9b7ab0d845312746"
format_stamp: "Formatted at 2025-06-24 14:20:26 on dist-test-slave-hs17"
I20250624 14:20:26.308792 7266 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a1845000ee254a6b9b7ab0d845312746"
format_stamp: "Formatted at 2025-06-24 14:20:26 on dist-test-slave-hs17"
I20250624 14:20:26.317518 7266 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250624 14:20:26.323982 7283 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:26.325162 7266 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250624 14:20:26.325508 7266 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a1845000ee254a6b9b7ab0d845312746"
format_stamp: "Formatted at 2025-06-24 14:20:26 on dist-test-slave-hs17"
I20250624 14:20:26.325840 7266 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:20:26.400619 7266 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:20:26.402163 7266 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:20:26.402596 7266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:20:26.405488 7266 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:20:26.410130 7266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:20:26.410372 7266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:26.410655 7266 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:20:26.410840 7266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:26.592531 7266 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.5:34865
I20250624 14:20:26.592741 7396 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.5:34865 every 8 connection(s)
I20250624 14:20:26.595311 7266 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250624 14:20:26.606029 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 7266
I20250624 14:20:26.606595 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250624 14:20:26.621959 7397 heartbeater.cc:344] Connected to a master server at 127.2.14.62:38655
I20250624 14:20:26.622426 7397 heartbeater.cc:461] Registering TS with master...
I20250624 14:20:26.623499 7397 heartbeater.cc:507] Master 127.2.14.62:38655 requested a full tablet report, sending...
I20250624 14:20:26.626003 6674 ts_manager.cc:194] Registered new tserver with Master: a1845000ee254a6b9b7ab0d845312746 (127.2.14.5:34865)
I20250624 14:20:26.627920 6674 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.5:42977
I20250624 14:20:26.630585 2104 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250624 14:20:26.674929 6674 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44378:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250624 14:20:26.790345 7064 tablet_service.cc:1468] Processing CreateTablet for tablet 13e4213513e74106b8b7c5911313b380 (DEFAULT_TABLE table=TestTable [id=45a603d562b64fbbb958fbad883b3636]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:20:26.791378 6930 tablet_service.cc:1468] Processing CreateTablet for tablet 13e4213513e74106b8b7c5911313b380 (DEFAULT_TABLE table=TestTable [id=45a603d562b64fbbb958fbad883b3636]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:20:26.793569 6930 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13e4213513e74106b8b7c5911313b380. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:20:26.796934 7331 tablet_service.cc:1468] Processing CreateTablet for tablet 13e4213513e74106b8b7c5911313b380 (DEFAULT_TABLE table=TestTable [id=45a603d562b64fbbb958fbad883b3636]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:20:26.793336 7064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13e4213513e74106b8b7c5911313b380. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:20:26.798774 7331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13e4213513e74106b8b7c5911313b380. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:20:26.829237 7416 tablet_bootstrap.cc:492] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba: Bootstrap starting.
I20250624 14:20:26.833909 7417 tablet_bootstrap.cc:492] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746: Bootstrap starting.
I20250624 14:20:26.838299 7416 tablet_bootstrap.cc:654] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba: Neither blocks nor log segments found. Creating new log.
I20250624 14:20:26.840852 7416 log.cc:826] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba: Log is configured to *not* fsync() on all Append() calls
I20250624 14:20:26.844316 7417 tablet_bootstrap.cc:654] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746: Neither blocks nor log segments found. Creating new log.
I20250624 14:20:26.848042 7417 log.cc:826] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746: Log is configured to *not* fsync() on all Append() calls
I20250624 14:20:26.853222 7419 tablet_bootstrap.cc:492] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be: Bootstrap starting.
I20250624 14:20:26.857430 7416 tablet_bootstrap.cc:492] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba: No bootstrap required, opened a new log
I20250624 14:20:26.858019 7416 ts_tablet_manager.cc:1397] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.000s
I20250624 14:20:26.859488 7417 tablet_bootstrap.cc:492] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746: No bootstrap required, opened a new log
I20250624 14:20:26.860105 7417 ts_tablet_manager.cc:1397] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.000s
I20250624 14:20:26.862643 7419 tablet_bootstrap.cc:654] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be: Neither blocks nor log segments found. Creating new log.
I20250624 14:20:26.865327 7419 log.cc:826] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be: Log is configured to *not* fsync() on all Append() calls
I20250624 14:20:26.877292 7419 tablet_bootstrap.cc:492] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be: No bootstrap required, opened a new log
I20250624 14:20:26.878077 7419 ts_tablet_manager.cc:1397] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.012s
I20250624 14:20:26.894207 7417 raft_consensus.cc:357] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fa16017d034c6792c347dc57d345ba" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 33661 } } peers { permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 } } peers { permanent_uuid: "a1845000ee254a6b9b7ab0d845312746" member_type: VOTER last_known_addr { host: "127.2.14.5" port: 34865 } }
I20250624 14:20:26.895473 7417 raft_consensus.cc:383] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:20:26.895882 7417 raft_consensus.cc:738] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1845000ee254a6b9b7ab0d845312746, State: Initialized, Role: FOLLOWER
I20250624 14:20:26.896936 7417 consensus_queue.cc:260] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fa16017d034c6792c347dc57d345ba" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 33661 } } peers { permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 } } peers { permanent_uuid: "a1845000ee254a6b9b7ab0d845312746" member_type: VOTER last_known_addr { host: "127.2.14.5" port: 34865 } }
I20250624 14:20:26.910156 7416 raft_consensus.cc:357] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fa16017d034c6792c347dc57d345ba" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 33661 } } peers { permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 } } peers { permanent_uuid: "a1845000ee254a6b9b7ab0d845312746" member_type: VOTER last_known_addr { host: "127.2.14.5" port: 34865 } }
I20250624 14:20:26.911231 7416 raft_consensus.cc:383] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:20:26.911604 7416 raft_consensus.cc:738] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26fa16017d034c6792c347dc57d345ba, State: Initialized, Role: FOLLOWER
I20250624 14:20:26.912885 7397 heartbeater.cc:499] Master 127.2.14.62:38655 was elected leader, sending a full tablet report...
I20250624 14:20:26.912580 7416 consensus_queue.cc:260] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fa16017d034c6792c347dc57d345ba" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 33661 } } peers { permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 } } peers { permanent_uuid: "a1845000ee254a6b9b7ab0d845312746" member_type: VOTER last_known_addr { host: "127.2.14.5" port: 34865 } }
I20250624 14:20:26.913650 7417 ts_tablet_manager.cc:1428] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746: Time spent starting tablet: real 0.053s user 0.031s sys 0.016s
I20250624 14:20:26.915975 7419 raft_consensus.cc:357] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fa16017d034c6792c347dc57d345ba" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 33661 } } peers { permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 } } peers { permanent_uuid: "a1845000ee254a6b9b7ab0d845312746" member_type: VOTER last_known_addr { host: "127.2.14.5" port: 34865 } }
I20250624 14:20:26.917066 7419 raft_consensus.cc:383] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:20:26.917421 7419 raft_consensus.cc:738] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9016a354df354ae9a10f8cb6ac60d0be, State: Initialized, Role: FOLLOWER
I20250624 14:20:26.918428 7419 consensus_queue.cc:260] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fa16017d034c6792c347dc57d345ba" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 33661 } } peers { permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 } } peers { permanent_uuid: "a1845000ee254a6b9b7ab0d845312746" member_type: VOTER last_known_addr { host: "127.2.14.5" port: 34865 } }
I20250624 14:20:26.956496 7416 ts_tablet_manager.cc:1428] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba: Time spent starting tablet: real 0.098s user 0.039s sys 0.037s
I20250624 14:20:26.959712 7419 ts_tablet_manager.cc:1428] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be: Time spent starting tablet: real 0.081s user 0.050s sys 0.024s
I20250624 14:20:26.973166 7422 raft_consensus.cc:491] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 14:20:26.973723 7422 raft_consensus.cc:513] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fa16017d034c6792c347dc57d345ba" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 33661 } } peers { permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 } } peers { permanent_uuid: "a1845000ee254a6b9b7ab0d845312746" member_type: VOTER last_known_addr { host: "127.2.14.5" port: 34865 } }
I20250624 14:20:26.976281 7422 leader_election.cc:290] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26fa16017d034c6792c347dc57d345ba (127.2.14.2:33661), 9016a354df354ae9a10f8cb6ac60d0be (127.2.14.3:41413)
I20250624 14:20:26.987017 6950 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13e4213513e74106b8b7c5911313b380" candidate_uuid: "a1845000ee254a6b9b7ab0d845312746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26fa16017d034c6792c347dc57d345ba" is_pre_election: true
I20250624 14:20:26.987897 6950 raft_consensus.cc:2466] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1845000ee254a6b9b7ab0d845312746 in term 0.
W20250624 14:20:26.988749 7130 tablet.cc:2378] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:20:26.989180 7084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13e4213513e74106b8b7c5911313b380" candidate_uuid: "a1845000ee254a6b9b7ab0d845312746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9016a354df354ae9a10f8cb6ac60d0be" is_pre_election: true
I20250624 14:20:26.989368 7286 leader_election.cc:304] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26fa16017d034c6792c347dc57d345ba, a1845000ee254a6b9b7ab0d845312746; no voters:
I20250624 14:20:26.989926 7084 raft_consensus.cc:2466] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1845000ee254a6b9b7ab0d845312746 in term 0.
I20250624 14:20:26.990239 7422 raft_consensus.cc:2802] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 14:20:26.990599 7422 raft_consensus.cc:491] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 14:20:26.990914 7422 raft_consensus.cc:3058] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:20:26.995519 7422 raft_consensus.cc:513] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fa16017d034c6792c347dc57d345ba" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 33661 } } peers { permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 } } peers { permanent_uuid: "a1845000ee254a6b9b7ab0d845312746" member_type: VOTER last_known_addr { host: "127.2.14.5" port: 34865 } }
I20250624 14:20:26.997026 7422 leader_election.cc:290] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [CANDIDATE]: Term 1 election: Requested vote from peers 26fa16017d034c6792c347dc57d345ba (127.2.14.2:33661), 9016a354df354ae9a10f8cb6ac60d0be (127.2.14.3:41413)
I20250624 14:20:26.997987 6950 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13e4213513e74106b8b7c5911313b380" candidate_uuid: "a1845000ee254a6b9b7ab0d845312746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26fa16017d034c6792c347dc57d345ba"
I20250624 14:20:26.998075 7084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13e4213513e74106b8b7c5911313b380" candidate_uuid: "a1845000ee254a6b9b7ab0d845312746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9016a354df354ae9a10f8cb6ac60d0be"
I20250624 14:20:26.998579 6950 raft_consensus.cc:3058] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:20:26.998580 7084 raft_consensus.cc:3058] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:20:27.005849 6950 raft_consensus.cc:2466] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1845000ee254a6b9b7ab0d845312746 in term 1.
I20250624 14:20:27.005859 7084 raft_consensus.cc:2466] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1845000ee254a6b9b7ab0d845312746 in term 1.
I20250624 14:20:27.006999 7286 leader_election.cc:304] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26fa16017d034c6792c347dc57d345ba, a1845000ee254a6b9b7ab0d845312746; no voters:
I20250624 14:20:27.007685 7422 raft_consensus.cc:2802] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:20:27.009370 7422 raft_consensus.cc:695] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [term 1 LEADER]: Becoming Leader. State: Replica: a1845000ee254a6b9b7ab0d845312746, State: Running, Role: LEADER
I20250624 14:20:27.010150 7422 consensus_queue.cc:237] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fa16017d034c6792c347dc57d345ba" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 33661 } } peers { permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 } } peers { permanent_uuid: "a1845000ee254a6b9b7ab0d845312746" member_type: VOTER last_known_addr { host: "127.2.14.5" port: 34865 } }
I20250624 14:20:27.022017 6672 catalog_manager.cc:5582] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1845000ee254a6b9b7ab0d845312746 (127.2.14.5). New cstate: current_term: 1 leader_uuid: "a1845000ee254a6b9b7ab0d845312746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fa16017d034c6792c347dc57d345ba" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 33661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1845000ee254a6b9b7ab0d845312746" member_type: VOTER last_known_addr { host: "127.2.14.5" port: 34865 } health_report { overall_health: HEALTHY } } }
I20250624 14:20:27.084036 2104 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250624 14:20:27.087589 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 26fa16017d034c6792c347dc57d345ba to finish bootstrapping
I20250624 14:20:27.101519 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9016a354df354ae9a10f8cb6ac60d0be to finish bootstrapping
W20250624 14:20:27.102982 7398 tablet.cc:2378] T 13e4213513e74106b8b7c5911313b380 P a1845000ee254a6b9b7ab0d845312746: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:20:27.113349 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a1845000ee254a6b9b7ab0d845312746 to finish bootstrapping
I20250624 14:20:27.125471 2104 test_util.cc:276] Using random seed: 128387783
I20250624 14:20:27.152004 2104 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250624 14:20:27.152963 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 7266
W20250624 14:20:27.155949 6996 tablet.cc:2378] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 14:20:27.181468 7433 negotiation.cc:337] Failed RPC negotiation. Trace:
0624 14:20:27.164519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.14.5:34865 (local address 127.0.0.1:52086)
0624 14:20:27.165495 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0624 14:20:27.165547 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0624 14:20:27.165877 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0624 14:20:27.180194 (+ 14317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.14.5:34865: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":87}
W20250624 14:20:27.194800 7430 meta_cache.cc:302] tablet 13e4213513e74106b8b7c5911313b380: replica a1845000ee254a6b9b7ab0d845312746 (127.2.14.5:34865) has failed: Network error: Client connection negotiation failed: client connection to 127.2.14.5:34865: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 14:20:27.217581 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:27.235728 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:27.257505 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:27.269533 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:27.298286 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:27.314285 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:27.344516 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:27.361661 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:27.397982 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:27.416725 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:27.459520 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:27.483184 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:27.532213 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:27.560202 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:27.620101 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:27.653353 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:27.723336 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:27.757385 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:27.829654 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:27.864974 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:27.949690 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:27.993548 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:28.086241 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:28.131798 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:28.226321 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:28.274238 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:28.326274 7430 meta_cache.cc:302] tablet 13e4213513e74106b8b7c5911313b380: replica a1845000ee254a6b9b7ab0d845312746 (127.2.14.5:34865) has failed: Network error: Client connection negotiation failed: client connection to 127.2.14.5:34865: connect: Connection refused (error 111)
W20250624 14:20:28.373622 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:28.424131 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:28.528831 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:28.579463 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:28.694515 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
W20250624 14:20:28.747968 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:28.862742 6910 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33052: Illegal state: replica 26fa16017d034c6792c347dc57d345ba is not leader of this config: current role FOLLOWER
I20250624 14:20:28.890862 7447 raft_consensus.cc:491] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 14:20:28.891386 7447 raft_consensus.cc:513] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fa16017d034c6792c347dc57d345ba" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 33661 } } peers { permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 } } peers { permanent_uuid: "a1845000ee254a6b9b7ab0d845312746" member_type: VOTER last_known_addr { host: "127.2.14.5" port: 34865 } }
I20250624 14:20:28.894347 7447 leader_election.cc:290] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9016a354df354ae9a10f8cb6ac60d0be (127.2.14.3:41413), a1845000ee254a6b9b7ab0d845312746 (127.2.14.5:34865)
W20250624 14:20:28.917580 7044 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42570: Illegal state: replica 9016a354df354ae9a10f8cb6ac60d0be is not leader of this config: current role FOLLOWER
W20250624 14:20:28.948565 6884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.14.5:34865: connect: Connection refused (error 111)
I20250624 14:20:28.953342 7084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13e4213513e74106b8b7c5911313b380" candidate_uuid: "26fa16017d034c6792c347dc57d345ba" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9016a354df354ae9a10f8cb6ac60d0be" is_pre_election: true
I20250624 14:20:28.954016 7084 raft_consensus.cc:2466] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26fa16017d034c6792c347dc57d345ba in term 1.
W20250624 14:20:28.955139 6884 leader_election.cc:336] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a1845000ee254a6b9b7ab0d845312746 (127.2.14.5:34865): Network error: Client connection negotiation failed: client connection to 127.2.14.5:34865: connect: Connection refused (error 111)
I20250624 14:20:28.956141 6884 leader_election.cc:304] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26fa16017d034c6792c347dc57d345ba, 9016a354df354ae9a10f8cb6ac60d0be; no voters: a1845000ee254a6b9b7ab0d845312746
I20250624 14:20:28.957132 7447 raft_consensus.cc:2802] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250624 14:20:28.957460 7447 raft_consensus.cc:491] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 14:20:28.957723 7447 raft_consensus.cc:3058] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [term 1 FOLLOWER]: Advancing to term 2
I20250624 14:20:28.962877 7447 raft_consensus.cc:513] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fa16017d034c6792c347dc57d345ba" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 33661 } } peers { permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 } } peers { permanent_uuid: "a1845000ee254a6b9b7ab0d845312746" member_type: VOTER last_known_addr { host: "127.2.14.5" port: 34865 } }
I20250624 14:20:28.964887 7447 leader_election.cc:290] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [CANDIDATE]: Term 2 election: Requested vote from peers 9016a354df354ae9a10f8cb6ac60d0be (127.2.14.3:41413), a1845000ee254a6b9b7ab0d845312746 (127.2.14.5:34865)
I20250624 14:20:28.967118 7084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13e4213513e74106b8b7c5911313b380" candidate_uuid: "26fa16017d034c6792c347dc57d345ba" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9016a354df354ae9a10f8cb6ac60d0be"
I20250624 14:20:28.967891 7084 raft_consensus.cc:3058] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be [term 1 FOLLOWER]: Advancing to term 2
I20250624 14:20:28.985172 7084 raft_consensus.cc:2466] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26fa16017d034c6792c347dc57d345ba in term 2.
I20250624 14:20:28.993196 6884 leader_election.cc:304] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26fa16017d034c6792c347dc57d345ba, 9016a354df354ae9a10f8cb6ac60d0be; no voters:
W20250624 14:20:28.993868 6884 leader_election.cc:336] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a1845000ee254a6b9b7ab0d845312746 (127.2.14.5:34865): Network error: Client connection negotiation failed: client connection to 127.2.14.5:34865: connect: Connection refused (error 111)
I20250624 14:20:28.993824 7447 raft_consensus.cc:2802] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [term 2 FOLLOWER]: Leader election won for term 2
I20250624 14:20:29.012809 7447 raft_consensus.cc:695] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [term 2 LEADER]: Becoming Leader. State: Replica: 26fa16017d034c6792c347dc57d345ba, State: Running, Role: LEADER
I20250624 14:20:29.014000 7447 consensus_queue.cc:237] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fa16017d034c6792c347dc57d345ba" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 33661 } } peers { permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 } } peers { permanent_uuid: "a1845000ee254a6b9b7ab0d845312746" member_type: VOTER last_known_addr { host: "127.2.14.5" port: 34865 } }
I20250624 14:20:29.028366 6672 catalog_manager.cc:5582] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba reported cstate change: term changed from 1 to 2, leader changed from a1845000ee254a6b9b7ab0d845312746 (127.2.14.5) to 26fa16017d034c6792c347dc57d345ba (127.2.14.2). New cstate: current_term: 2 leader_uuid: "26fa16017d034c6792c347dc57d345ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26fa16017d034c6792c347dc57d345ba" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 33661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1845000ee254a6b9b7ab0d845312746" member_type: VOTER last_known_addr { host: "127.2.14.5" port: 34865 } health_report { overall_health: UNKNOWN } } }
I20250624 14:20:29.047159 7084 raft_consensus.cc:1273] T 13e4213513e74106b8b7c5911313b380 P 9016a354df354ae9a10f8cb6ac60d0be [term 2 FOLLOWER]: Refusing update from remote peer 26fa16017d034c6792c347dc57d345ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250624 14:20:29.049338 6884 consensus_peers.cc:487] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba -> Peer a1845000ee254a6b9b7ab0d845312746 (127.2.14.5:34865): Couldn't send request to peer a1845000ee254a6b9b7ab0d845312746. Status: Network error: Client connection negotiation failed: client connection to 127.2.14.5:34865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250624 14:20:29.050736 7447 consensus_queue.cc:1035] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9016a354df354ae9a10f8cb6ac60d0be" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 41413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 14:20:29.089824 7460 mvcc.cc:204] Tried to move back new op lower bound from 7171173699753717760 to 7171173699651485696. Current Snapshot: MvccSnapshot[applied={T|T < 7171173699753717760}]
I20250624 14:20:29.416986 7197 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 14:20:29.445992 6797 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 14:20:29.446174 6930 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 14:20:29.501459 7064 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 14:20:31.199913 7197 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 14:20:31.251863 6930 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250624 14:20:31.260430 7064 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 14:20:31.288375 6797 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250624 14:20:31.611776 6884 consensus_peers.cc:487] T 13e4213513e74106b8b7c5911313b380 P 26fa16017d034c6792c347dc57d345ba -> Peer a1845000ee254a6b9b7ab0d845312746 (127.2.14.5:34865): Couldn't send request to peer a1845000ee254a6b9b7ab0d845312746. Status: Network error: Client connection negotiation failed: client connection to 127.2.14.5:34865: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250624 14:20:33.673765 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 6733
I20250624 14:20:33.700970 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 6866
I20250624 14:20:33.744439 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 6999
I20250624 14:20:33.780344 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 7133
I20250624 14:20:33.806669 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 6641
2025-06-24T14:20:33Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (18868 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (18868 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250624 14:20:33.868935 2104 test_util.cc:276] Using random seed: 135131240
I20250624 14:20:33.873515 2104 ts_itest-base.cc:115] Starting cluster with:
I20250624 14:20:33.873705 2104 ts_itest-base.cc:116] --------------
I20250624 14:20:33.873879 2104 ts_itest-base.cc:117] 1 tablet servers
I20250624 14:20:33.874054 2104 ts_itest-base.cc:118] 1 replicas per TS
I20250624 14:20:33.874207 2104 ts_itest-base.cc:119] --------------
2025-06-24T14:20:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T14:20:33Z Disabled control of system clock
I20250624 14:20:33.929548 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:37727
--webserver_interface=127.2.14.62
--webserver_port=0
--builtin_ntp_servers=127.2.14.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.14.62:37727 with env {}
W20250624 14:20:34.255878 7546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:20:34.256526 7546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:20:34.257014 7546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:20:34.289322 7546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 14:20:34.289649 7546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:20:34.289924 7546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 14:20:34.290165 7546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 14:20:34.326766 7546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.14.62:37727
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:37727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:20:34.328243 7546 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:20:34.330159 7546 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:20:34.345808 7552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:34.346166 7553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:34.347182 7555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:34.347514 7546 server_base.cc:1048] running on GCE node
I20250624 14:20:35.546079 7546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:20:35.549049 7546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:20:35.550434 7546 hybrid_clock.cc:648] HybridClock initialized: now 1750774835550388 us; error 61 us; skew 500 ppm
I20250624 14:20:35.551317 7546 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:20:35.564049 7546 webserver.cc:469] Webserver started at http://127.2.14.62:37749/ using document root <none> and password file <none>
I20250624 14:20:35.565142 7546 fs_manager.cc:362] Metadata directory not provided
I20250624 14:20:35.565382 7546 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:20:35.565940 7546 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:20:35.570755 7546 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4d43aca496c14aeda66083fb6837b177"
format_stamp: "Formatted at 2025-06-24 14:20:35 on dist-test-slave-hs17"
I20250624 14:20:35.572021 7546 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4d43aca496c14aeda66083fb6837b177"
format_stamp: "Formatted at 2025-06-24 14:20:35 on dist-test-slave-hs17"
I20250624 14:20:35.579891 7546 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20250624 14:20:35.585917 7562 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:35.587002 7546 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250624 14:20:35.587344 7546 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4d43aca496c14aeda66083fb6837b177"
format_stamp: "Formatted at 2025-06-24 14:20:35 on dist-test-slave-hs17"
I20250624 14:20:35.587723 7546 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:20:35.648065 7546 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:20:35.649621 7546 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:20:35.650118 7546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:20:35.724464 7546 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.62:37727
I20250624 14:20:35.724540 7613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.62:37727 every 8 connection(s)
I20250624 14:20:35.727298 7546 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250624 14:20:35.732594 7614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:20:35.733155 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 7546
I20250624 14:20:35.733667 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250624 14:20:35.759769 7614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177: Bootstrap starting.
I20250624 14:20:35.765436 7614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177: Neither blocks nor log segments found. Creating new log.
I20250624 14:20:35.767164 7614 log.cc:826] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177: Log is configured to *not* fsync() on all Append() calls
I20250624 14:20:35.772078 7614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177: No bootstrap required, opened a new log
I20250624 14:20:35.790555 7614 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d43aca496c14aeda66083fb6837b177" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37727 } }
I20250624 14:20:35.791249 7614 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:20:35.791528 7614 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d43aca496c14aeda66083fb6837b177, State: Initialized, Role: FOLLOWER
I20250624 14:20:35.792269 7614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d43aca496c14aeda66083fb6837b177" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37727 } }
I20250624 14:20:35.792783 7614 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:20:35.793079 7614 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:20:35.793390 7614 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:20:35.797878 7614 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d43aca496c14aeda66083fb6837b177" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37727 } }
I20250624 14:20:35.798615 7614 leader_election.cc:304] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d43aca496c14aeda66083fb6837b177; no voters:
I20250624 14:20:35.800380 7614 leader_election.cc:290] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 14:20:35.801156 7619 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:20:35.803376 7619 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [term 1 LEADER]: Becoming Leader. State: Replica: 4d43aca496c14aeda66083fb6837b177, State: Running, Role: LEADER
I20250624 14:20:35.804108 7619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d43aca496c14aeda66083fb6837b177" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37727 } }
I20250624 14:20:35.804555 7614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 14:20:35.817706 7621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d43aca496c14aeda66083fb6837b177. Latest consensus state: current_term: 1 leader_uuid: "4d43aca496c14aeda66083fb6837b177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d43aca496c14aeda66083fb6837b177" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37727 } } }
I20250624 14:20:35.818774 7621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [sys.catalog]: This master's current role is: LEADER
I20250624 14:20:35.822561 7620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d43aca496c14aeda66083fb6837b177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d43aca496c14aeda66083fb6837b177" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 37727 } } }
I20250624 14:20:35.825032 7620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177 [sys.catalog]: This master's current role is: LEADER
I20250624 14:20:35.832718 7628 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 14:20:35.844426 7628 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 14:20:35.859401 7628 catalog_manager.cc:1349] Generated new cluster ID: 6d3dd044a3e34ac0a51fbef61ccba188
I20250624 14:20:35.859772 7628 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 14:20:35.875727 7628 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 14:20:35.877334 7628 catalog_manager.cc:1506] Loading token signing keys...
I20250624 14:20:35.893538 7628 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4d43aca496c14aeda66083fb6837b177: Generated new TSK 0
I20250624 14:20:35.894521 7628 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 14:20:35.917742 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.1:0
--local_ip_for_outbound_sockets=127.2.14.1
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:37727
--builtin_ntp_servers=127.2.14.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250624 14:20:36.278132 7638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:20:36.278735 7638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:20:36.279268 7638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:20:36.313094 7638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:20:36.313992 7638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.1
I20250624 14:20:36.350679 7638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:37727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.1
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:20:36.352188 7638 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:20:36.353950 7638 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:20:36.373479 7645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:36.375222 7644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:36.373518 7647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:37.554996 7646 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 14:20:37.555435 7638 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:20:37.559345 7638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:20:37.562207 7638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:20:37.563675 7638 hybrid_clock.cc:648] HybridClock initialized: now 1750774837563609 us; error 90 us; skew 500 ppm
I20250624 14:20:37.564523 7638 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:20:37.571825 7638 webserver.cc:469] Webserver started at http://127.2.14.1:40041/ using document root <none> and password file <none>
I20250624 14:20:37.572871 7638 fs_manager.cc:362] Metadata directory not provided
I20250624 14:20:37.573122 7638 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:20:37.573601 7638 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:20:37.578295 7638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "63bd410ee99c4d1785c429d0b36f4ba4"
format_stamp: "Formatted at 2025-06-24 14:20:37 on dist-test-slave-hs17"
I20250624 14:20:37.579483 7638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "63bd410ee99c4d1785c429d0b36f4ba4"
format_stamp: "Formatted at 2025-06-24 14:20:37 on dist-test-slave-hs17"
I20250624 14:20:37.587409 7638 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250624 14:20:37.593971 7654 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:37.595463 7638 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20250624 14:20:37.595883 7638 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "63bd410ee99c4d1785c429d0b36f4ba4"
format_stamp: "Formatted at 2025-06-24 14:20:37 on dist-test-slave-hs17"
I20250624 14:20:37.596233 7638 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:20:37.646270 7638 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:20:37.647897 7638 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:20:37.648336 7638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:20:37.651453 7638 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:20:37.655870 7638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:20:37.656088 7638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:37.656371 7638 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:20:37.656519 7638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:37.796170 7638 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.1:40231
I20250624 14:20:37.796288 7766 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.1:40231 every 8 connection(s)
I20250624 14:20:37.798780 7638 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250624 14:20:37.802038 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 7638
I20250624 14:20:37.802588 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750774691568470-2104-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250624 14:20:37.820735 7767 heartbeater.cc:344] Connected to a master server at 127.2.14.62:37727
I20250624 14:20:37.821215 7767 heartbeater.cc:461] Registering TS with master...
I20250624 14:20:37.822495 7767 heartbeater.cc:507] Master 127.2.14.62:37727 requested a full tablet report, sending...
I20250624 14:20:37.825039 7578 ts_manager.cc:194] Registered new tserver with Master: 63bd410ee99c4d1785c429d0b36f4ba4 (127.2.14.1:40231)
I20250624 14:20:37.827021 7578 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.1:56971
I20250624 14:20:37.836237 2104 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250624 14:20:37.866962 7578 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35262:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250624 14:20:37.927031 7702 tablet_service.cc:1468] Processing CreateTablet for tablet 0d4c7a46e2e04a55a6a21dd0bec84722 (DEFAULT_TABLE table=TestTable [id=998e7482978a4cfe812b4261cb803f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:20:37.928598 7702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d4c7a46e2e04a55a6a21dd0bec84722. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:20:37.949808 7782 tablet_bootstrap.cc:492] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4: Bootstrap starting.
I20250624 14:20:37.955412 7782 tablet_bootstrap.cc:654] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4: Neither blocks nor log segments found. Creating new log.
I20250624 14:20:37.957383 7782 log.cc:826] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4: Log is configured to *not* fsync() on all Append() calls
I20250624 14:20:37.962450 7782 tablet_bootstrap.cc:492] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4: No bootstrap required, opened a new log
I20250624 14:20:37.962924 7782 ts_tablet_manager.cc:1397] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20250624 14:20:37.981081 7782 raft_consensus.cc:357] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bd410ee99c4d1785c429d0b36f4ba4" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 40231 } }
I20250624 14:20:37.981714 7782 raft_consensus.cc:383] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:20:37.981942 7782 raft_consensus.cc:738] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63bd410ee99c4d1785c429d0b36f4ba4, State: Initialized, Role: FOLLOWER
I20250624 14:20:37.982646 7782 consensus_queue.cc:260] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bd410ee99c4d1785c429d0b36f4ba4" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 40231 } }
I20250624 14:20:37.983180 7782 raft_consensus.cc:397] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:20:37.983441 7782 raft_consensus.cc:491] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:20:37.983785 7782 raft_consensus.cc:3058] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:20:37.988641 7782 raft_consensus.cc:513] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bd410ee99c4d1785c429d0b36f4ba4" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 40231 } }
I20250624 14:20:37.989372 7782 leader_election.cc:304] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63bd410ee99c4d1785c429d0b36f4ba4; no voters:
I20250624 14:20:37.991096 7782 leader_election.cc:290] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 14:20:37.991918 7784 raft_consensus.cc:2802] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:20:37.994295 7784 raft_consensus.cc:695] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4 [term 1 LEADER]: Becoming Leader. State: Replica: 63bd410ee99c4d1785c429d0b36f4ba4, State: Running, Role: LEADER
I20250624 14:20:37.994657 7767 heartbeater.cc:499] Master 127.2.14.62:37727 was elected leader, sending a full tablet report...
I20250624 14:20:37.995261 7784 consensus_queue.cc:237] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bd410ee99c4d1785c429d0b36f4ba4" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 40231 } }
I20250624 14:20:37.995711 7782 ts_tablet_manager.cc:1428] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4: Time spent starting tablet: real 0.033s user 0.027s sys 0.008s
I20250624 14:20:38.008399 7578 catalog_manager.cc:5582] T 0d4c7a46e2e04a55a6a21dd0bec84722 P 63bd410ee99c4d1785c429d0b36f4ba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63bd410ee99c4d1785c429d0b36f4ba4 (127.2.14.1). New cstate: current_term: 1 leader_uuid: "63bd410ee99c4d1785c429d0b36f4ba4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bd410ee99c4d1785c429d0b36f4ba4" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 40231 } health_report { overall_health: HEALTHY } } }
I20250624 14:20:38.032400 2104 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250624 14:20:38.035400 2104 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 63bd410ee99c4d1785c429d0b36f4ba4 to finish bootstrapping
I20250624 14:20:40.783447 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 7638
I20250624 14:20:40.812285 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 7546
2025-06-24T14:20:40Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7001 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7001 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250624 14:20:40.870857 2104 test_util.cc:276] Using random seed: 142133162
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (13 ms)
[----------] 1 test from ListTableCliParamTest (13 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-24T14:20:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T14:20:40Z Disabled control of system clock
I20250624 14:20:40.939941 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:33331
--webserver_interface=127.2.14.62
--webserver_port=0
--builtin_ntp_servers=127.2.14.20:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.14.62:33331 with env {}
W20250624 14:20:41.255797 7810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:20:41.256402 7810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:20:41.256876 7810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:20:41.288199 7810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 14:20:41.288551 7810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:20:41.288820 7810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 14:20:41.289062 7810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 14:20:41.324759 7810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.14.62:33331
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:33331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.14.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:20:41.326157 7810 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:20:41.327895 7810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:20:41.342327 7817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:41.343448 7816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:41.344135 7819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:41.344206 7810 server_base.cc:1048] running on GCE node
I20250624 14:20:42.534796 7810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:20:42.537624 7810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:20:42.539036 7810 hybrid_clock.cc:648] HybridClock initialized: now 1750774842538982 us; error 54 us; skew 500 ppm
I20250624 14:20:42.539968 7810 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:20:42.547153 7810 webserver.cc:469] Webserver started at http://127.2.14.62:44109/ using document root <none> and password file <none>
I20250624 14:20:42.548298 7810 fs_manager.cc:362] Metadata directory not provided
I20250624 14:20:42.548542 7810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:20:42.549044 7810 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:20:42.553911 7810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data/instance:
uuid: "b8b4def1b91248e0adbc8c08e857dfc4"
format_stamp: "Formatted at 2025-06-24 14:20:42 on dist-test-slave-hs17"
I20250624 14:20:42.555119 7810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal/instance:
uuid: "b8b4def1b91248e0adbc8c08e857dfc4"
format_stamp: "Formatted at 2025-06-24 14:20:42 on dist-test-slave-hs17"
I20250624 14:20:42.563409 7810 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20250624 14:20:42.569494 7827 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:42.570631 7810 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250624 14:20:42.571004 7810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal
uuid: "b8b4def1b91248e0adbc8c08e857dfc4"
format_stamp: "Formatted at 2025-06-24 14:20:42 on dist-test-slave-hs17"
I20250624 14:20:42.571331 7810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:20:42.632617 7810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:20:42.634096 7810 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:20:42.634508 7810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:20:42.705794 7810 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.62:33331
I20250624 14:20:42.705905 7878 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.62:33331 every 8 connection(s)
I20250624 14:20:42.708624 7810 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data/info.pb
I20250624 14:20:42.713774 7879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:20:42.714785 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 7810
I20250624 14:20:42.715373 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal/instance
I20250624 14:20:42.734791 7879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4: Bootstrap starting.
I20250624 14:20:42.741604 7879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4: Neither blocks nor log segments found. Creating new log.
I20250624 14:20:42.743413 7879 log.cc:826] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4: Log is configured to *not* fsync() on all Append() calls
I20250624 14:20:42.748227 7879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4: No bootstrap required, opened a new log
I20250624 14:20:42.767010 7879 raft_consensus.cc:357] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b4def1b91248e0adbc8c08e857dfc4" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 33331 } }
I20250624 14:20:42.767778 7879 raft_consensus.cc:383] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:20:42.767995 7879 raft_consensus.cc:738] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b4def1b91248e0adbc8c08e857dfc4, State: Initialized, Role: FOLLOWER
I20250624 14:20:42.768693 7879 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b4def1b91248e0adbc8c08e857dfc4" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 33331 } }
I20250624 14:20:42.769209 7879 raft_consensus.cc:397] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:20:42.769440 7879 raft_consensus.cc:491] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:20:42.769709 7879 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:20:42.774323 7879 raft_consensus.cc:513] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b4def1b91248e0adbc8c08e857dfc4" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 33331 } }
I20250624 14:20:42.775117 7879 leader_election.cc:304] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8b4def1b91248e0adbc8c08e857dfc4; no voters:
I20250624 14:20:42.776818 7879 leader_election.cc:290] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 14:20:42.777581 7884 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:20:42.779829 7884 raft_consensus.cc:695] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [term 1 LEADER]: Becoming Leader. State: Replica: b8b4def1b91248e0adbc8c08e857dfc4, State: Running, Role: LEADER
I20250624 14:20:42.780607 7884 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b4def1b91248e0adbc8c08e857dfc4" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 33331 } }
I20250624 14:20:42.781744 7879 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 14:20:42.793221 7885 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8b4def1b91248e0adbc8c08e857dfc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b4def1b91248e0adbc8c08e857dfc4" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 33331 } } }
I20250624 14:20:42.793401 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b4def1b91248e0adbc8c08e857dfc4. Latest consensus state: current_term: 1 leader_uuid: "b8b4def1b91248e0adbc8c08e857dfc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b4def1b91248e0adbc8c08e857dfc4" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 33331 } } }
I20250624 14:20:42.794111 7885 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [sys.catalog]: This master's current role is: LEADER
I20250624 14:20:42.794315 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4 [sys.catalog]: This master's current role is: LEADER
I20250624 14:20:42.798177 7894 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 14:20:42.810449 7894 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 14:20:42.826191 7894 catalog_manager.cc:1349] Generated new cluster ID: eee0a7260ddc4b889cbe15d49f49cc62
I20250624 14:20:42.826517 7894 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 14:20:42.871265 7894 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 14:20:42.873473 7894 catalog_manager.cc:1506] Loading token signing keys...
I20250624 14:20:42.893934 7894 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b8b4def1b91248e0adbc8c08e857dfc4: Generated new TSK 0
I20250624 14:20:42.894907 7894 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 14:20:42.905122 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.1:0
--local_ip_for_outbound_sockets=127.2.14.1
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:33331
--builtin_ntp_servers=127.2.14.20:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250624 14:20:43.246856 7903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:20:43.247395 7903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:20:43.247943 7903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:20:43.279947 7903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:20:43.280860 7903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.1
I20250624 14:20:43.317780 7903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.14.1
--webserver_port=0
--tserver_master_addrs=127.2.14.62:33331
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.1
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:20:43.319229 7903 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:20:43.321094 7903 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:20:43.340483 7910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:43.342695 7909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:43.343183 7912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:43.345827 7903 server_base.cc:1048] running on GCE node
I20250624 14:20:44.531183 7903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:20:44.534081 7903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:20:44.535666 7903 hybrid_clock.cc:648] HybridClock initialized: now 1750774844535616 us; error 51 us; skew 500 ppm
I20250624 14:20:44.536782 7903 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:20:44.545055 7903 webserver.cc:469] Webserver started at http://127.2.14.1:37825/ using document root <none> and password file <none>
I20250624 14:20:44.546401 7903 fs_manager.cc:362] Metadata directory not provided
I20250624 14:20:44.546692 7903 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:20:44.547291 7903 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:20:44.554225 7903 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data/instance:
uuid: "c15e00f3c26d478b9de63617d296dee0"
format_stamp: "Formatted at 2025-06-24 14:20:44 on dist-test-slave-hs17"
I20250624 14:20:44.555847 7903 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/wal/instance:
uuid: "c15e00f3c26d478b9de63617d296dee0"
format_stamp: "Formatted at 2025-06-24 14:20:44 on dist-test-slave-hs17"
I20250624 14:20:44.565439 7903 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.002s sys 0.007s
I20250624 14:20:44.573689 7919 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:44.574982 7903 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20250624 14:20:44.575398 7903 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/wal
uuid: "c15e00f3c26d478b9de63617d296dee0"
format_stamp: "Formatted at 2025-06-24 14:20:44 on dist-test-slave-hs17"
I20250624 14:20:44.575887 7903 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:20:44.644161 7903 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:20:44.645948 7903 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:20:44.646488 7903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:20:44.649772 7903 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:20:44.654521 7903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:20:44.654789 7903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:44.655085 7903 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:20:44.655256 7903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:44.799515 7903 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.1:46525
I20250624 14:20:44.799682 8031 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.1:46525 every 8 connection(s)
I20250624 14:20:44.802183 7903 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data/info.pb
I20250624 14:20:44.805004 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 7903
I20250624 14:20:44.805550 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/wal/instance
I20250624 14:20:44.813563 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.2:0
--local_ip_for_outbound_sockets=127.2.14.2
--webserver_interface=127.2.14.2
--webserver_port=0
--tserver_master_addrs=127.2.14.62:33331
--builtin_ntp_servers=127.2.14.20:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 14:20:44.827687 8032 heartbeater.cc:344] Connected to a master server at 127.2.14.62:33331
I20250624 14:20:44.828143 8032 heartbeater.cc:461] Registering TS with master...
I20250624 14:20:44.829217 8032 heartbeater.cc:507] Master 127.2.14.62:33331 requested a full tablet report, sending...
I20250624 14:20:44.832279 7844 ts_manager.cc:194] Registered new tserver with Master: c15e00f3c26d478b9de63617d296dee0 (127.2.14.1:46525)
I20250624 14:20:44.835283 7844 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.1:52539
W20250624 14:20:45.127885 8036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:20:45.128372 8036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:20:45.128831 8036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:20:45.161293 8036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:20:45.162119 8036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.2
I20250624 14:20:45.196949 8036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.14.2
--webserver_port=0
--tserver_master_addrs=127.2.14.62:33331
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.2
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:20:45.198231 8036 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:20:45.200009 8036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:20:45.217835 8042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:45.839713 8032 heartbeater.cc:499] Master 127.2.14.62:33331 was elected leader, sending a full tablet report...
W20250624 14:20:45.217941 8045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:45.217948 8043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:46.387403 8044 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 14:20:46.387531 8036 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:20:46.391851 8036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:20:46.394193 8036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:20:46.395597 8036 hybrid_clock.cc:648] HybridClock initialized: now 1750774846395487 us; error 75 us; skew 500 ppm
I20250624 14:20:46.396430 8036 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:20:46.407327 8036 webserver.cc:469] Webserver started at http://127.2.14.2:41187/ using document root <none> and password file <none>
I20250624 14:20:46.408313 8036 fs_manager.cc:362] Metadata directory not provided
I20250624 14:20:46.408540 8036 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:20:46.409004 8036 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:20:46.413591 8036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data/instance:
uuid: "45a4ffd4b59c43d997373faec35cc264"
format_stamp: "Formatted at 2025-06-24 14:20:46 on dist-test-slave-hs17"
I20250624 14:20:46.414705 8036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/wal/instance:
uuid: "45a4ffd4b59c43d997373faec35cc264"
format_stamp: "Formatted at 2025-06-24 14:20:46 on dist-test-slave-hs17"
I20250624 14:20:46.421826 8036 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250624 14:20:46.427838 8052 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:46.428915 8036 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250624 14:20:46.429247 8036 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/wal
uuid: "45a4ffd4b59c43d997373faec35cc264"
format_stamp: "Formatted at 2025-06-24 14:20:46 on dist-test-slave-hs17"
I20250624 14:20:46.429585 8036 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:20:46.488654 8036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:20:46.490161 8036 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:20:46.490614 8036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:20:46.493163 8036 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:20:46.497419 8036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:20:46.497642 8036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:46.497912 8036 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:20:46.498077 8036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:46.631346 8036 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.2:36527
I20250624 14:20:46.631451 8164 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.2:36527 every 8 connection(s)
I20250624 14:20:46.633962 8036 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data/info.pb
I20250624 14:20:46.638682 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 8036
I20250624 14:20:46.639218 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/wal/instance
I20250624 14:20:46.645385 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.3:0
--local_ip_for_outbound_sockets=127.2.14.3
--webserver_interface=127.2.14.3
--webserver_port=0
--tserver_master_addrs=127.2.14.62:33331
--builtin_ntp_servers=127.2.14.20:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 14:20:46.655737 8165 heartbeater.cc:344] Connected to a master server at 127.2.14.62:33331
I20250624 14:20:46.656168 8165 heartbeater.cc:461] Registering TS with master...
I20250624 14:20:46.657188 8165 heartbeater.cc:507] Master 127.2.14.62:33331 requested a full tablet report, sending...
I20250624 14:20:46.659228 7844 ts_manager.cc:194] Registered new tserver with Master: 45a4ffd4b59c43d997373faec35cc264 (127.2.14.2:36527)
I20250624 14:20:46.660490 7844 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.2:56299
W20250624 14:20:46.958643 8169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:20:46.959177 8169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:20:46.959710 8169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:20:46.996176 8169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:20:46.997053 8169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.3
I20250624 14:20:47.032552 8169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.14.3
--webserver_port=0
--tserver_master_addrs=127.2.14.62:33331
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.3
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:20:47.033977 8169 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:20:47.035696 8169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:20:47.053145 8176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:47.663646 8165 heartbeater.cc:499] Master 127.2.14.62:33331 was elected leader, sending a full tablet report...
W20250624 14:20:47.053134 8178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:47.053382 8175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:47.055696 8169 server_base.cc:1048] running on GCE node
I20250624 14:20:48.220836 8169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:20:48.223171 8169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:20:48.224524 8169 hybrid_clock.cc:648] HybridClock initialized: now 1750774848224487 us; error 58 us; skew 500 ppm
I20250624 14:20:48.225338 8169 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:20:48.231524 8169 webserver.cc:469] Webserver started at http://127.2.14.3:37043/ using document root <none> and password file <none>
I20250624 14:20:48.232518 8169 fs_manager.cc:362] Metadata directory not provided
I20250624 14:20:48.232718 8169 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:20:48.233196 8169 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:20:48.237694 8169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data/instance:
uuid: "b25f654e2f274423880aee74a30d5dab"
format_stamp: "Formatted at 2025-06-24 14:20:48 on dist-test-slave-hs17"
I20250624 14:20:48.238793 8169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/wal/instance:
uuid: "b25f654e2f274423880aee74a30d5dab"
format_stamp: "Formatted at 2025-06-24 14:20:48 on dist-test-slave-hs17"
I20250624 14:20:48.246088 8169 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250624 14:20:48.251645 8185 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:48.252856 8169 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250624 14:20:48.253175 8169 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/wal
uuid: "b25f654e2f274423880aee74a30d5dab"
format_stamp: "Formatted at 2025-06-24 14:20:48 on dist-test-slave-hs17"
I20250624 14:20:48.253515 8169 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:20:48.332463 8169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:20:48.333964 8169 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:20:48.334403 8169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:20:48.337018 8169 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:20:48.341146 8169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 14:20:48.341434 8169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:48.341701 8169 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 14:20:48.341874 8169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:48.477944 8169 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.3:40469
I20250624 14:20:48.478050 8297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.3:40469 every 8 connection(s)
I20250624 14:20:48.480454 8169 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data/info.pb
I20250624 14:20:48.490784 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 8169
I20250624 14:20:48.491297 2104 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/wal/instance
I20250624 14:20:48.502928 8298 heartbeater.cc:344] Connected to a master server at 127.2.14.62:33331
I20250624 14:20:48.503332 8298 heartbeater.cc:461] Registering TS with master...
I20250624 14:20:48.504376 8298 heartbeater.cc:507] Master 127.2.14.62:33331 requested a full tablet report, sending...
I20250624 14:20:48.506472 7844 ts_manager.cc:194] Registered new tserver with Master: b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469)
I20250624 14:20:48.507743 7844 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.3:38097
I20250624 14:20:48.512133 2104 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 14:20:48.542193 2104 test_util.cc:276] Using random seed: 149804506
I20250624 14:20:48.583415 7844 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39260:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250624 14:20:48.585914 7844 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 14:20:48.644707 8233 tablet_service.cc:1468] Processing CreateTablet for tablet 37c156433d6d45af8a62d6f6b858c0ed (DEFAULT_TABLE table=pre_rebuild [id=0d130625b16542518106c41039fbd3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:20:48.645222 8100 tablet_service.cc:1468] Processing CreateTablet for tablet 37c156433d6d45af8a62d6f6b858c0ed (DEFAULT_TABLE table=pre_rebuild [id=0d130625b16542518106c41039fbd3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:20:48.647172 8100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c156433d6d45af8a62d6f6b858c0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:20:48.646620 7967 tablet_service.cc:1468] Processing CreateTablet for tablet 37c156433d6d45af8a62d6f6b858c0ed (DEFAULT_TABLE table=pre_rebuild [id=0d130625b16542518106c41039fbd3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:20:48.647248 8233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c156433d6d45af8a62d6f6b858c0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:20:48.648578 7967 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c156433d6d45af8a62d6f6b858c0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:20:48.676578 8322 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: Bootstrap starting.
I20250624 14:20:48.680428 8324 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: Bootstrap starting.
I20250624 14:20:48.680451 8323 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Bootstrap starting.
I20250624 14:20:48.685214 8322 tablet_bootstrap.cc:654] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: Neither blocks nor log segments found. Creating new log.
I20250624 14:20:48.686093 8324 tablet_bootstrap.cc:654] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: Neither blocks nor log segments found. Creating new log.
I20250624 14:20:48.687983 8322 log.cc:826] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: Log is configured to *not* fsync() on all Append() calls
I20250624 14:20:48.688419 8324 log.cc:826] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: Log is configured to *not* fsync() on all Append() calls
I20250624 14:20:48.688881 8323 tablet_bootstrap.cc:654] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Neither blocks nor log segments found. Creating new log.
I20250624 14:20:48.691700 8323 log.cc:826] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Log is configured to *not* fsync() on all Append() calls
I20250624 14:20:48.694353 8322 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: No bootstrap required, opened a new log
I20250624 14:20:48.694639 8324 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: No bootstrap required, opened a new log
I20250624 14:20:48.694914 8322 ts_tablet_manager.cc:1397] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.000s
I20250624 14:20:48.695137 8324 ts_tablet_manager.cc:1397] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.005s
I20250624 14:20:48.696800 8323 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: No bootstrap required, opened a new log
I20250624 14:20:48.697556 8323 ts_tablet_manager.cc:1397] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20250624 14:20:48.721401 8323 raft_consensus.cc:357] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:20:48.721935 8322 raft_consensus.cc:357] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:20:48.722528 8323 raft_consensus.cc:383] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:20:48.722849 8323 raft_consensus.cc:738] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a4ffd4b59c43d997373faec35cc264, State: Initialized, Role: FOLLOWER
I20250624 14:20:48.722906 8322 raft_consensus.cc:383] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:20:48.723212 8322 raft_consensus.cc:738] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25f654e2f274423880aee74a30d5dab, State: Initialized, Role: FOLLOWER
I20250624 14:20:48.723795 8323 consensus_queue.cc:260] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:20:48.724252 8322 consensus_queue.cc:260] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:20:48.730504 8324 raft_consensus.cc:357] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:20:48.732043 8324 raft_consensus.cc:383] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:20:48.732488 8324 raft_consensus.cc:738] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15e00f3c26d478b9de63617d296dee0, State: Initialized, Role: FOLLOWER
I20250624 14:20:48.733878 8324 consensus_queue.cc:260] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:20:48.737764 8323 ts_tablet_manager.cc:1428] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Time spent starting tablet: real 0.040s user 0.036s sys 0.004s
I20250624 14:20:48.738575 8298 heartbeater.cc:499] Master 127.2.14.62:33331 was elected leader, sending a full tablet report...
I20250624 14:20:48.739117 8322 ts_tablet_manager.cc:1428] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: Time spent starting tablet: real 0.044s user 0.025s sys 0.016s
I20250624 14:20:48.740804 8324 ts_tablet_manager.cc:1428] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: Time spent starting tablet: real 0.045s user 0.039s sys 0.000s
W20250624 14:20:48.810987 8033 tablet.cc:2378] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:20:48.831188 8330 raft_consensus.cc:491] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 14:20:48.831722 8330 raft_consensus.cc:513] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:20:48.834103 8330 leader_election.cc:290] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45a4ffd4b59c43d997373faec35cc264 (127.2.14.2:36527), b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469)
I20250624 14:20:48.845285 8253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37c156433d6d45af8a62d6f6b858c0ed" candidate_uuid: "c15e00f3c26d478b9de63617d296dee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25f654e2f274423880aee74a30d5dab" is_pre_election: true
I20250624 14:20:48.845291 8120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37c156433d6d45af8a62d6f6b858c0ed" candidate_uuid: "c15e00f3c26d478b9de63617d296dee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a4ffd4b59c43d997373faec35cc264" is_pre_election: true
I20250624 14:20:48.845980 8120 raft_consensus.cc:2466] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c15e00f3c26d478b9de63617d296dee0 in term 0.
I20250624 14:20:48.845985 8253 raft_consensus.cc:2466] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c15e00f3c26d478b9de63617d296dee0 in term 0.
I20250624 14:20:48.847088 7923 leader_election.cc:304] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b25f654e2f274423880aee74a30d5dab, c15e00f3c26d478b9de63617d296dee0; no voters:
I20250624 14:20:48.847811 8330 raft_consensus.cc:2802] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 14:20:48.848140 8330 raft_consensus.cc:491] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 14:20:48.848351 8330 raft_consensus.cc:3058] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:20:48.852650 8330 raft_consensus.cc:513] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:20:48.853981 8330 leader_election.cc:290] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [CANDIDATE]: Term 1 election: Requested vote from peers 45a4ffd4b59c43d997373faec35cc264 (127.2.14.2:36527), b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469)
I20250624 14:20:48.854843 8120 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37c156433d6d45af8a62d6f6b858c0ed" candidate_uuid: "c15e00f3c26d478b9de63617d296dee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a4ffd4b59c43d997373faec35cc264"
I20250624 14:20:48.855027 8253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37c156433d6d45af8a62d6f6b858c0ed" candidate_uuid: "c15e00f3c26d478b9de63617d296dee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25f654e2f274423880aee74a30d5dab"
I20250624 14:20:48.855269 8120 raft_consensus.cc:3058] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:20:48.855504 8253 raft_consensus.cc:3058] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:20:48.859716 8120 raft_consensus.cc:2466] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c15e00f3c26d478b9de63617d296dee0 in term 1.
I20250624 14:20:48.860110 8253 raft_consensus.cc:2466] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c15e00f3c26d478b9de63617d296dee0 in term 1.
I20250624 14:20:48.860742 7920 leader_election.cc:304] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a4ffd4b59c43d997373faec35cc264, c15e00f3c26d478b9de63617d296dee0; no voters:
I20250624 14:20:48.861544 8330 raft_consensus.cc:2802] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:20:48.863067 8330 raft_consensus.cc:695] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 1 LEADER]: Becoming Leader. State: Replica: c15e00f3c26d478b9de63617d296dee0, State: Running, Role: LEADER
I20250624 14:20:48.863888 8330 consensus_queue.cc:237] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:20:48.875424 7841 catalog_manager.cc:5582] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c15e00f3c26d478b9de63617d296dee0 (127.2.14.1). New cstate: current_term: 1 leader_uuid: "c15e00f3c26d478b9de63617d296dee0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } health_report { overall_health: HEALTHY } } }
W20250624 14:20:48.891095 8166 tablet.cc:2378] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 14:20:48.985349 8299 tablet.cc:2378] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:20:49.040076 8253 raft_consensus.cc:1273] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab [term 1 FOLLOWER]: Refusing update from remote peer c15e00f3c26d478b9de63617d296dee0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 14:20:49.040833 8120 raft_consensus.cc:1273] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 1 FOLLOWER]: Refusing update from remote peer c15e00f3c26d478b9de63617d296dee0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 14:20:49.042158 8330 consensus_queue.cc:1035] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250624 14:20:49.043292 8333 consensus_queue.cc:1035] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250624 14:20:54.461349 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 7810
W20250624 14:20:54.905542 8374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:20:54.906345 8374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:20:54.941943 8374 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250624 14:20:55.164532 8032 heartbeater.cc:646] Failed to heartbeat to 127.2.14.62:33331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.14.62:33331: connect: Connection refused (error 111)
W20250624 14:20:55.208990 8298 heartbeater.cc:646] Failed to heartbeat to 127.2.14.62:33331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.14.62:33331: connect: Connection refused (error 111)
W20250624 14:20:55.215817 8165 heartbeater.cc:646] Failed to heartbeat to 127.2.14.62:33331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.14.62:33331: connect: Connection refused (error 111)
W20250624 14:20:56.334946 8374 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.347s user 0.526s sys 0.791s
W20250624 14:20:56.335326 8374 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.348s user 0.526s sys 0.791s
I20250624 14:20:56.491331 8374 minidump.cc:252] Setting minidump size limit to 20M
I20250624 14:20:56.494159 8374 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:20:56.495954 8374 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:20:56.510095 8410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:56.511529 8411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:56.597836 8374 server_base.cc:1048] running on GCE node
W20250624 14:20:56.597947 8413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:20:56.599099 8374 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250624 14:20:56.599617 8374 hybrid_clock.cc:648] HybridClock initialized: now 1750774856599589 us; error 321307 us; skew 500 ppm
I20250624 14:20:56.600543 8374 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:20:56.605198 8374 webserver.cc:469] Webserver started at http://0.0.0.0:44977/ using document root <none> and password file <none>
I20250624 14:20:56.606051 8374 fs_manager.cc:362] Metadata directory not provided
I20250624 14:20:56.606271 8374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:20:56.606696 8374 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 14:20:56.610976 8374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data/instance:
uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75"
format_stamp: "Formatted at 2025-06-24 14:20:56 on dist-test-slave-hs17"
I20250624 14:20:56.612093 8374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal/instance:
uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75"
format_stamp: "Formatted at 2025-06-24 14:20:56 on dist-test-slave-hs17"
I20250624 14:20:56.618275 8374 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20250624 14:20:56.623219 8418 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:56.624476 8374 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250624 14:20:56.624794 8374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal
uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75"
format_stamp: "Formatted at 2025-06-24 14:20:56 on dist-test-slave-hs17"
I20250624 14:20:56.625104 8374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:20:56.726475 8374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:20:56.728089 8374 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:20:56.728546 8374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:20:56.733510 8374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:20:56.748924 8374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75: Bootstrap starting.
I20250624 14:20:56.754202 8374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75: Neither blocks nor log segments found. Creating new log.
I20250624 14:20:56.756145 8374 log.cc:826] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75: Log is configured to *not* fsync() on all Append() calls
I20250624 14:20:56.760659 8374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75: No bootstrap required, opened a new log
I20250624 14:20:56.778436 8374 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" member_type: VOTER }
I20250624 14:20:56.778985 8374 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:20:56.779242 8374 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdddb09ff5343ba92a6b6fd03bc4b75, State: Initialized, Role: FOLLOWER
I20250624 14:20:56.780040 8374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" member_type: VOTER }
I20250624 14:20:56.780506 8374 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:20:56.780779 8374 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:20:56.781078 8374 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:20:56.785733 8374 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" member_type: VOTER }
I20250624 14:20:56.786489 8374 leader_election.cc:304] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cdddb09ff5343ba92a6b6fd03bc4b75; no voters:
I20250624 14:20:56.788455 8374 leader_election.cc:290] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 14:20:56.788689 8425 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:20:56.791595 8425 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 1 LEADER]: Becoming Leader. State: Replica: 6cdddb09ff5343ba92a6b6fd03bc4b75, State: Running, Role: LEADER
I20250624 14:20:56.792532 8425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" member_type: VOTER }
I20250624 14:20:56.799021 8427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cdddb09ff5343ba92a6b6fd03bc4b75. Latest consensus state: current_term: 1 leader_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" member_type: VOTER } }
I20250624 14:20:56.799487 8426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" member_type: VOTER } }
I20250624 14:20:56.800194 8426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [sys.catalog]: This master's current role is: LEADER
I20250624 14:20:56.799980 8427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [sys.catalog]: This master's current role is: LEADER
I20250624 14:20:56.813683 8374 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75: stopping tablet replica
I20250624 14:20:56.814414 8374 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 1 LEADER]: Raft consensus shutting down.
I20250624 14:20:56.814987 8374 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 14:20:56.817808 8374 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250624 14:20:56.818367 8374 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250624 14:20:56.847663 8374 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250624 14:20:57.881253 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 7903
I20250624 14:20:57.926501 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 8036
I20250624 14:20:57.961206 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 8169
I20250624 14:20:58.001860 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:33331
--webserver_interface=127.2.14.62
--webserver_port=44109
--builtin_ntp_servers=127.2.14.20:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.14.62:33331 with env {}
W20250624 14:20:58.323549 8435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:20:58.324218 8435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:20:58.324712 8435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:20:58.356195 8435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 14:20:58.356575 8435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:20:58.356832 8435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 14:20:58.357074 8435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 14:20:58.392825 8435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.14.62:33331
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.14.62:33331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.14.62
--webserver_port=44109
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:20:58.394212 8435 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:20:58.395920 8435 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:20:58.410229 8442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:58.410259 8441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:58.412142 8444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:20:59.625886 8443 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1213 milliseconds
I20250624 14:20:59.626020 8435 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:20:59.627301 8435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:20:59.630657 8435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:20:59.632242 8435 hybrid_clock.cc:648] HybridClock initialized: now 1750774859632185 us; error 72 us; skew 500 ppm
I20250624 14:20:59.633136 8435 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:20:59.639869 8435 webserver.cc:469] Webserver started at http://127.2.14.62:44109/ using document root <none> and password file <none>
I20250624 14:20:59.640941 8435 fs_manager.cc:362] Metadata directory not provided
I20250624 14:20:59.641206 8435 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:20:59.650575 8435 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20250624 14:20:59.657655 8451 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:20:59.658900 8435 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250624 14:20:59.659305 8435 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal
uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75"
format_stamp: "Formatted at 2025-06-24 14:20:56 on dist-test-slave-hs17"
I20250624 14:20:59.661468 8435 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:20:59.720062 8435 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:20:59.721604 8435 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:20:59.722033 8435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:20:59.796676 8435 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.62:33331
I20250624 14:20:59.796809 8502 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.62:33331 every 8 connection(s)
I20250624 14:20:59.799719 8435 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data/info.pb
I20250624 14:20:59.805794 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 8435
I20250624 14:20:59.807952 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.1:46525
--local_ip_for_outbound_sockets=127.2.14.1
--tserver_master_addrs=127.2.14.62:33331
--webserver_port=37825
--webserver_interface=127.2.14.1
--builtin_ntp_servers=127.2.14.20:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 14:20:59.813447 8503 sys_catalog.cc:263] Verifying existing consensus state
I20250624 14:20:59.831056 8503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75: Bootstrap starting.
I20250624 14:20:59.842264 8503 log.cc:826] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75: Log is configured to *not* fsync() on all Append() calls
I20250624 14:20:59.855475 8503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:20:59.856371 8503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75: Bootstrap complete.
I20250624 14:20:59.877803 8503 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 33331 } }
I20250624 14:20:59.878501 8503 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdddb09ff5343ba92a6b6fd03bc4b75, State: Initialized, Role: FOLLOWER
I20250624 14:20:59.879331 8503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 33331 } }
I20250624 14:20:59.879933 8503 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 14:20:59.880220 8503 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 14:20:59.880517 8503 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 1 FOLLOWER]: Advancing to term 2
I20250624 14:20:59.884877 8503 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 33331 } }
I20250624 14:20:59.885676 8503 leader_election.cc:304] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cdddb09ff5343ba92a6b6fd03bc4b75; no voters:
I20250624 14:20:59.888023 8503 leader_election.cc:290] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250624 14:20:59.888432 8507 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 14:20:59.891909 8507 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [term 2 LEADER]: Becoming Leader. State: Replica: 6cdddb09ff5343ba92a6b6fd03bc4b75, State: Running, Role: LEADER
I20250624 14:20:59.892830 8507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 33331 } }
I20250624 14:20:59.893633 8503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 14:20:59.904242 8508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 33331 } } }
I20250624 14:20:59.904295 8509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cdddb09ff5343ba92a6b6fd03bc4b75. Latest consensus state: current_term: 2 leader_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdddb09ff5343ba92a6b6fd03bc4b75" member_type: VOTER last_known_addr { host: "127.2.14.62" port: 33331 } } }
I20250624 14:20:59.905045 8508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [sys.catalog]: This master's current role is: LEADER
I20250624 14:20:59.905197 8509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75 [sys.catalog]: This master's current role is: LEADER
I20250624 14:20:59.916631 8513 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 14:20:59.930155 8513 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=f15ee8ca224543e1bc742c213bbd2ca1]
I20250624 14:20:59.938966 8513 tablet_loader.cc:96] loaded metadata for tablet 37c156433d6d45af8a62d6f6b858c0ed (table pre_rebuild [id=f15ee8ca224543e1bc742c213bbd2ca1])
I20250624 14:20:59.940891 8513 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 14:20:59.972728 8513 catalog_manager.cc:1349] Generated new cluster ID: dfaf7977253648cda41c9862c37aa1f5
I20250624 14:20:59.973150 8513 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 14:21:00.021585 8513 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 14:21:00.023068 8513 catalog_manager.cc:1506] Loading token signing keys...
I20250624 14:21:00.039953 8513 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75: Generated new TSK 0
I20250624 14:21:00.040912 8513 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250624 14:21:00.191351 8505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:21:00.192137 8505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:21:00.192968 8505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:21:00.228049 8505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:21:00.229195 8505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.1
I20250624 14:21:00.264284 8505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.1:46525
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.14.1
--webserver_port=37825
--tserver_master_addrs=127.2.14.62:33331
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.1
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:21:00.265679 8505 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:21:00.267374 8505 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:21:00.284924 8532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:21:00.287036 8505 server_base.cc:1048] running on GCE node
W20250624 14:21:00.287333 8534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:21:00.286338 8531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:21:01.505101 8505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:21:01.508008 8505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:21:01.509559 8505 hybrid_clock.cc:648] HybridClock initialized: now 1750774861509486 us; error 85 us; skew 500 ppm
I20250624 14:21:01.510687 8505 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:21:01.531883 8505 webserver.cc:469] Webserver started at http://127.2.14.1:37825/ using document root <none> and password file <none>
I20250624 14:21:01.533154 8505 fs_manager.cc:362] Metadata directory not provided
I20250624 14:21:01.533443 8505 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:21:01.544292 8505 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20250624 14:21:01.550426 8541 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:21:01.551831 8505 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20250624 14:21:01.552243 8505 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/wal
uuid: "c15e00f3c26d478b9de63617d296dee0"
format_stamp: "Formatted at 2025-06-24 14:20:44 on dist-test-slave-hs17"
I20250624 14:21:01.555045 8505 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:21:01.622488 8505 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:21:01.624465 8505 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:21:01.624974 8505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:21:01.627701 8505 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:21:01.633895 8548 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250624 14:21:01.645319 8505 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 14:21:01.645612 8505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250624 14:21:01.645915 8505 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 14:21:01.650957 8505 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 14:21:01.651175 8505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250624 14:21:01.651759 8548 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: Bootstrap starting.
I20250624 14:21:01.877846 8505 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.1:46525
I20250624 14:21:01.878060 8655 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.1:46525 every 8 connection(s)
I20250624 14:21:01.881953 8505 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data/info.pb
I20250624 14:21:01.887552 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 8505
I20250624 14:21:01.889595 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.2:36527
--local_ip_for_outbound_sockets=127.2.14.2
--tserver_master_addrs=127.2.14.62:33331
--webserver_port=41187
--webserver_interface=127.2.14.2
--builtin_ntp_servers=127.2.14.20:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 14:21:01.947274 8656 heartbeater.cc:344] Connected to a master server at 127.2.14.62:33331
I20250624 14:21:01.947876 8656 heartbeater.cc:461] Registering TS with master...
I20250624 14:21:01.949262 8656 heartbeater.cc:507] Master 127.2.14.62:33331 requested a full tablet report, sending...
I20250624 14:21:01.954810 8468 ts_manager.cc:194] Registered new tserver with Master: c15e00f3c26d478b9de63617d296dee0 (127.2.14.1:46525)
I20250624 14:21:01.962872 8468 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.1:47543
I20250624 14:21:02.005554 8548 log.cc:826] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: Log is configured to *not* fsync() on all Append() calls
W20250624 14:21:02.282087 8660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:21:02.282634 8660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:21:02.283233 8660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:21:02.315480 8660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:21:02.316393 8660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.2
I20250624 14:21:02.352337 8660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.2:36527
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.14.2
--webserver_port=41187
--tserver_master_addrs=127.2.14.62:33331
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.2
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:21:02.353717 8660 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:21:02.355508 8660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:21:02.373911 8668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:21:02.967308 8656 heartbeater.cc:499] Master 127.2.14.62:33331 was elected leader, sending a full tablet report...
W20250624 14:21:02.380471 8671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:21:02.373916 8669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:21:03.687445 8670 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1311 milliseconds
I20250624 14:21:03.687608 8660 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 14:21:03.688958 8660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:21:03.695935 8660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:21:03.697466 8660 hybrid_clock.cc:648] HybridClock initialized: now 1750774863697411 us; error 74 us; skew 500 ppm
I20250624 14:21:03.698627 8660 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:21:03.706406 8660 webserver.cc:469] Webserver started at http://127.2.14.2:41187/ using document root <none> and password file <none>
I20250624 14:21:03.707417 8660 fs_manager.cc:362] Metadata directory not provided
I20250624 14:21:03.707697 8660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:21:03.716462 8660 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.004s
I20250624 14:21:03.721712 8678 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:21:03.722913 8660 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250624 14:21:03.723245 8660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/wal
uuid: "45a4ffd4b59c43d997373faec35cc264"
format_stamp: "Formatted at 2025-06-24 14:20:46 on dist-test-slave-hs17"
I20250624 14:21:03.725348 8660 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:21:03.788754 8660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:21:03.790268 8660 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:21:03.790737 8660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:21:03.793495 8660 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:21:03.801048 8685 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250624 14:21:03.809309 8660 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 14:21:03.809574 8660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.000s sys 0.003s
I20250624 14:21:03.809883 8660 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 14:21:03.814916 8660 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 14:21:03.815263 8660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.002s
I20250624 14:21:03.815641 8685 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Bootstrap starting.
I20250624 14:21:04.030725 8660 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.2:36527
I20250624 14:21:04.030875 8791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.2:36527 every 8 connection(s)
I20250624 14:21:04.034596 8660 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data/info.pb
I20250624 14:21:04.044193 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 8660
I20250624 14:21:04.046548 2104 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
/tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.14.3:40469
--local_ip_for_outbound_sockets=127.2.14.3
--tserver_master_addrs=127.2.14.62:33331
--webserver_port=37043
--webserver_interface=127.2.14.3
--builtin_ntp_servers=127.2.14.20:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250624 14:21:04.082396 8792 heartbeater.cc:344] Connected to a master server at 127.2.14.62:33331
I20250624 14:21:04.082967 8792 heartbeater.cc:461] Registering TS with master...
I20250624 14:21:04.084300 8792 heartbeater.cc:507] Master 127.2.14.62:33331 requested a full tablet report, sending...
I20250624 14:21:04.088673 8468 ts_manager.cc:194] Registered new tserver with Master: 45a4ffd4b59c43d997373faec35cc264 (127.2.14.2:36527)
I20250624 14:21:04.091899 8468 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.2:56971
I20250624 14:21:04.158012 8685 log.cc:826] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Log is configured to *not* fsync() on all Append() calls
W20250624 14:21:04.539669 8796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 14:21:04.540347 8796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 14:21:04.541110 8796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 14:21:04.597198 8796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 14:21:04.598601 8796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.14.3
I20250624 14:21:04.660526 8796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.14.20:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.14.3:40469
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.14.3
--webserver_port=37043
--tserver_master_addrs=127.2.14.62:33331
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.14.3
--log_dir=/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 14:14:57 UTC on 24a791456cd2
build id 6749
TSAN enabled
I20250624 14:21:04.662513 8796 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 14:21:04.664897 8796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 14:21:04.689327 8803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:21:05.087435 8548 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:21:05.088969 8548 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: Bootstrap complete.
I20250624 14:21:05.091495 8548 ts_tablet_manager.cc:1397] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: Time spent bootstrapping tablet: real 3.440s user 3.370s sys 0.055s
I20250624 14:21:05.096853 8792 heartbeater.cc:499] Master 127.2.14.62:33331 was elected leader, sending a full tablet report...
I20250624 14:21:05.116861 8548 raft_consensus.cc:357] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:21:05.121667 8548 raft_consensus.cc:738] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15e00f3c26d478b9de63617d296dee0, State: Initialized, Role: FOLLOWER
I20250624 14:21:05.123215 8548 consensus_queue.cc:260] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:21:05.138942 8548 ts_tablet_manager.cc:1428] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: Time spent starting tablet: real 0.047s user 0.032s sys 0.012s
W20250624 14:21:06.082562 8802 debug-util.cc:398] Leaking SignalData structure 0x7b0800000a80 after lost signal to thread 8796
W20250624 14:21:06.391510 8796 thread.cc:640] GCE (cloud detector) Time spent creating pthread: real 1.704s user 0.577s sys 1.024s
W20250624 14:21:04.691184 8804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 14:21:06.392227 8796 thread.cc:606] GCE (cloud detector) Time spent starting thread: real 1.705s user 0.578s sys 1.024s
W20250624 14:21:06.403096 8809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 14:21:06.403373 8796 server_base.cc:1048] running on GCE node
I20250624 14:21:06.404948 8796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 14:21:06.407887 8796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 14:21:06.409461 8796 hybrid_clock.cc:648] HybridClock initialized: now 1750774866409402 us; error 51 us; skew 500 ppm
I20250624 14:21:06.410672 8796 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 14:21:06.419378 8796 webserver.cc:469] Webserver started at http://127.2.14.3:37043/ using document root <none> and password file <none>
I20250624 14:21:06.420902 8796 fs_manager.cc:362] Metadata directory not provided
I20250624 14:21:06.421226 8796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 14:21:06.431034 8796 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20250624 14:21:06.436697 8814 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 14:21:06.438016 8796 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20250624 14:21:06.438375 8796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/wal
uuid: "b25f654e2f274423880aee74a30d5dab"
format_stamp: "Formatted at 2025-06-24 14:20:48 on dist-test-slave-hs17"
I20250624 14:21:06.440562 8796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 14:21:06.443167 8815 raft_consensus.cc:491] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 14:21:06.443746 8815 raft_consensus.cc:513] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:21:06.446905 8815 leader_election.cc:290] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45a4ffd4b59c43d997373faec35cc264 (127.2.14.2:36527), b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469)
W20250624 14:21:06.466856 8545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.14.3:40469: connect: Connection refused (error 111)
W20250624 14:21:06.478415 8545 leader_election.cc:336] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469): Network error: Client connection negotiation failed: client connection to 127.2.14.3:40469: connect: Connection refused (error 111)
I20250624 14:21:06.484468 8747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37c156433d6d45af8a62d6f6b858c0ed" candidate_uuid: "c15e00f3c26d478b9de63617d296dee0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "45a4ffd4b59c43d997373faec35cc264" is_pre_election: true
W20250624 14:21:06.499047 8542 leader_election.cc:343] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45a4ffd4b59c43d997373faec35cc264 (127.2.14.2:36527): Illegal state: must be running to vote when last-logged opid is not known
I20250624 14:21:06.499797 8542 leader_election.cc:304] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c15e00f3c26d478b9de63617d296dee0; no voters: 45a4ffd4b59c43d997373faec35cc264, b25f654e2f274423880aee74a30d5dab
I20250624 14:21:06.500738 8815 raft_consensus.cc:2747] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250624 14:21:06.526517 8796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 14:21:06.528162 8796 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 14:21:06.528676 8796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 14:21:06.531991 8796 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 14:21:06.539192 8826 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250624 14:21:06.547592 8796 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250624 14:21:06.547863 8796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.002s sys 0.001s
I20250624 14:21:06.548202 8796 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250624 14:21:06.553323 8796 ts_tablet_manager.cc:610] Registered 1 tablets
I20250624 14:21:06.553630 8796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250624 14:21:06.554077 8826 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: Bootstrap starting.
I20250624 14:21:06.606281 8685 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:21:06.607853 8685 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Bootstrap complete.
I20250624 14:21:06.610031 8685 ts_tablet_manager.cc:1397] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Time spent bootstrapping tablet: real 2.795s user 2.716s sys 0.064s
I20250624 14:21:06.619652 8685 raft_consensus.cc:357] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:21:06.623601 8685 raft_consensus.cc:738] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a4ffd4b59c43d997373faec35cc264, State: Initialized, Role: FOLLOWER
I20250624 14:21:06.624886 8685 consensus_queue.cc:260] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:21:06.628643 8685 ts_tablet_manager.cc:1428] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20250624 14:21:06.730631 8796 rpc_server.cc:307] RPC server started. Bound to: 127.2.14.3:40469
I20250624 14:21:06.730810 8933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.14.3:40469 every 8 connection(s)
I20250624 14:21:06.734331 8796 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data/info.pb
I20250624 14:21:06.742453 2104 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu as pid 8796
I20250624 14:21:06.769605 8934 heartbeater.cc:344] Connected to a master server at 127.2.14.62:33331
I20250624 14:21:06.770138 8934 heartbeater.cc:461] Registering TS with master...
I20250624 14:21:06.771375 8934 heartbeater.cc:507] Master 127.2.14.62:33331 requested a full tablet report, sending...
I20250624 14:21:06.775067 8468 ts_manager.cc:194] Registered new tserver with Master: b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469)
I20250624 14:21:06.777192 8468 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.14.3:48959
I20250624 14:21:06.787205 2104 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 14:21:06.874254 8826 log.cc:826] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: Log is configured to *not* fsync() on all Append() calls
I20250624 14:21:07.780895 8934 heartbeater.cc:499] Master 127.2.14.62:33331 was elected leader, sending a full tablet report...
I20250624 14:21:08.001535 8946 raft_consensus.cc:491] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 14:21:08.001997 8946 raft_consensus.cc:513] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:21:08.004297 8946 leader_election.cc:290] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469), c15e00f3c26d478b9de63617d296dee0 (127.2.14.1:46525)
I20250624 14:21:08.026657 8611 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37c156433d6d45af8a62d6f6b858c0ed" candidate_uuid: "45a4ffd4b59c43d997373faec35cc264" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "c15e00f3c26d478b9de63617d296dee0" is_pre_election: true
I20250624 14:21:08.027385 8611 raft_consensus.cc:2466] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45a4ffd4b59c43d997373faec35cc264 in term 1.
I20250624 14:21:08.029312 8679 leader_election.cc:304] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a4ffd4b59c43d997373faec35cc264, c15e00f3c26d478b9de63617d296dee0; no voters:
I20250624 14:21:08.030349 8946 raft_consensus.cc:2802] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250624 14:21:08.030727 8946 raft_consensus.cc:491] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 14:21:08.031064 8946 raft_consensus.cc:3058] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 1 FOLLOWER]: Advancing to term 2
I20250624 14:21:08.040778 8946 raft_consensus.cc:513] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:21:08.030229 8889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37c156433d6d45af8a62d6f6b858c0ed" candidate_uuid: "45a4ffd4b59c43d997373faec35cc264" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "b25f654e2f274423880aee74a30d5dab" is_pre_election: true
W20250624 14:21:08.043437 8682 leader_election.cc:343] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469): Illegal state: must be running to vote when last-logged opid is not known
I20250624 14:21:08.044090 8889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37c156433d6d45af8a62d6f6b858c0ed" candidate_uuid: "45a4ffd4b59c43d997373faec35cc264" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "b25f654e2f274423880aee74a30d5dab"
W20250624 14:21:08.045464 8682 leader_election.cc:343] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469): Illegal state: must be running to vote when last-logged opid is not known
I20250624 14:21:08.045907 8611 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37c156433d6d45af8a62d6f6b858c0ed" candidate_uuid: "45a4ffd4b59c43d997373faec35cc264" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "c15e00f3c26d478b9de63617d296dee0"
I20250624 14:21:08.046062 8946 leader_election.cc:290] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [CANDIDATE]: Term 2 election: Requested vote from peers b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469), c15e00f3c26d478b9de63617d296dee0 (127.2.14.1:46525)
I20250624 14:21:08.046448 8611 raft_consensus.cc:3058] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 1 FOLLOWER]: Advancing to term 2
I20250624 14:21:08.052954 8611 raft_consensus.cc:2466] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45a4ffd4b59c43d997373faec35cc264 in term 2.
I20250624 14:21:08.054121 8679 leader_election.cc:304] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45a4ffd4b59c43d997373faec35cc264, c15e00f3c26d478b9de63617d296dee0; no voters: b25f654e2f274423880aee74a30d5dab
I20250624 14:21:08.054895 8946 raft_consensus.cc:2802] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 14:21:08.056663 8946 raft_consensus.cc:695] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 2 LEADER]: Becoming Leader. State: Replica: 45a4ffd4b59c43d997373faec35cc264, State: Running, Role: LEADER
I20250624 14:21:08.057752 8946 consensus_queue.cc:237] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:21:08.070700 8468 catalog_manager.cc:5582] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 reported cstate change: term changed from 0 to 2, leader changed from <none> to 45a4ffd4b59c43d997373faec35cc264 (127.2.14.2), VOTER 45a4ffd4b59c43d997373faec35cc264 (127.2.14.2) added, VOTER b25f654e2f274423880aee74a30d5dab (127.2.14.3) added, VOTER c15e00f3c26d478b9de63617d296dee0 (127.2.14.1) added. New cstate: current_term: 2 leader_uuid: "45a4ffd4b59c43d997373faec35cc264" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } health_report { overall_health: UNKNOWN } } }
W20250624 14:21:08.506537 8682 consensus_peers.cc:487] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 -> Peer b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469): Couldn't send request to peer b25f654e2f274423880aee74a30d5dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250624 14:21:08.510905 8611 raft_consensus.cc:1273] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 2 FOLLOWER]: Refusing update from remote peer 45a4ffd4b59c43d997373faec35cc264: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250624 14:21:08.512449 8946 consensus_queue.cc:1035] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250624 14:21:08.603040 8747 consensus_queue.cc:237] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:21:08.608971 8611 raft_consensus.cc:1273] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 2 FOLLOWER]: Refusing update from remote peer 45a4ffd4b59c43d997373faec35cc264: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250624 14:21:08.610225 8946 consensus_queue.cc:1035] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250624 14:21:08.617448 8946 raft_consensus.cc:2953] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER b25f654e2f274423880aee74a30d5dab (127.2.14.3) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } } }
I20250624 14:21:08.620352 8610 raft_consensus.cc:2953] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER b25f654e2f274423880aee74a30d5dab (127.2.14.3) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } } }
I20250624 14:21:08.628816 8452 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 37c156433d6d45af8a62d6f6b858c0ed with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250624 14:21:08.632376 8468 catalog_manager.cc:5582] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 reported cstate change: config changed from index -1 to 207, VOTER b25f654e2f274423880aee74a30d5dab (127.2.14.3) evicted. New cstate: current_term: 2 leader_uuid: "45a4ffd4b59c43d997373faec35cc264" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } } }
W20250624 14:21:08.636126 2104 scanner-internal.cc:458] Time spent opening tablet: real 1.818s user 0.008s sys 0.000s
I20250624 14:21:08.665540 8747 consensus_queue.cc:237] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } }
I20250624 14:21:08.668598 8951 raft_consensus.cc:2953] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER c15e00f3c26d478b9de63617d296dee0 (127.2.14.1) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } }
I20250624 14:21:08.689275 8452 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 37c156433d6d45af8a62d6f6b858c0ed with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250624 14:21:08.695226 8468 catalog_manager.cc:5582] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 reported cstate change: config changed from index 207 to 208, VOTER c15e00f3c26d478b9de63617d296dee0 (127.2.14.1) evicted. New cstate: current_term: 2 leader_uuid: "45a4ffd4b59c43d997373faec35cc264" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } health_report { overall_health: HEALTHY } } }
I20250624 14:21:08.769461 8869 tablet_service.cc:1515] Processing DeleteTablet for tablet 37c156433d6d45af8a62d6f6b858c0ed with delete_type TABLET_DATA_TOMBSTONED (TS b25f654e2f274423880aee74a30d5dab not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:40228
I20250624 14:21:08.778106 8591 tablet_service.cc:1515] Processing DeleteTablet for tablet 37c156433d6d45af8a62d6f6b858c0ed with delete_type TABLET_DATA_TOMBSTONED (TS c15e00f3c26d478b9de63617d296dee0 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:45256
W20250624 14:21:08.787806 8455 catalog_manager.cc:4908] TS b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469): delete failed for tablet 37c156433d6d45af8a62d6f6b858c0ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37c156433d6d45af8a62d6f6b858c0ed already in progress: opening tablet
I20250624 14:21:08.800889 8974 tablet_replica.cc:331] stopping tablet replica
I20250624 14:21:08.801874 8974 raft_consensus.cc:2241] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250624 14:21:08.802675 8974 raft_consensus.cc:2270] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250624 14:21:08.863624 8974 ts_tablet_manager.cc:1905] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250624 14:21:08.885692 8974 ts_tablet_manager.cc:1918] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250624 14:21:08.886217 8974 log.cc:1199] T 37c156433d6d45af8a62d6f6b858c0ed P c15e00f3c26d478b9de63617d296dee0: Deleting WAL directory at /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/wal/wals/37c156433d6d45af8a62d6f6b858c0ed
I20250624 14:21:08.888182 8452 catalog_manager.cc:4928] TS c15e00f3c26d478b9de63617d296dee0 (127.2.14.1:46525): tablet 37c156433d6d45af8a62d6f6b858c0ed (table pre_rebuild [id=f15ee8ca224543e1bc742c213bbd2ca1]) successfully deleted
I20250624 14:21:09.270172 8826 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 14:21:09.271378 8826 tablet_bootstrap.cc:492] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: Bootstrap complete.
I20250624 14:21:09.273502 8826 ts_tablet_manager.cc:1397] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: Time spent bootstrapping tablet: real 2.720s user 2.572s sys 0.044s
I20250624 14:21:09.283169 8826 raft_consensus.cc:357] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:21:09.286518 8826 raft_consensus.cc:738] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25f654e2f274423880aee74a30d5dab, State: Initialized, Role: FOLLOWER
I20250624 14:21:09.287518 8826 consensus_queue.cc:260] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } }
I20250624 14:21:09.291360 8826 ts_tablet_manager.cc:1428] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20250624 14:21:09.296887 8869 tablet_service.cc:1515] Processing DeleteTablet for tablet 37c156433d6d45af8a62d6f6b858c0ed with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:40228
I20250624 14:21:09.304230 8983 tablet_replica.cc:331] stopping tablet replica
I20250624 14:21:09.305068 8983 raft_consensus.cc:2241] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 14:21:09.305624 8983 raft_consensus.cc:2270] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 14:21:09.338090 8983 ts_tablet_manager.cc:1905] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250624 14:21:09.358405 8983 ts_tablet_manager.cc:1918] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250624 14:21:09.358884 8983 log.cc:1199] T 37c156433d6d45af8a62d6f6b858c0ed P b25f654e2f274423880aee74a30d5dab: Deleting WAL directory at /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/wal/wals/37c156433d6d45af8a62d6f6b858c0ed
I20250624 14:21:09.360713 8455 catalog_manager.cc:4928] TS b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469): tablet 37c156433d6d45af8a62d6f6b858c0ed (table pre_rebuild [id=f15ee8ca224543e1bc742c213bbd2ca1]) successfully deleted
I20250624 14:21:09.561623 8869 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 14:21:09.561666 8591 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 14:21:09.562440 8727 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
6cdddb09ff5343ba92a6b6fd03bc4b75 | 127.2.14.62:33331 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.14.20:34391 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
45a4ffd4b59c43d997373faec35cc264 | 127.2.14.2:36527 | HEALTHY | <none> | 1 | 0
b25f654e2f274423880aee74a30d5dab | 127.2.14.3:40469 | HEALTHY | <none> | 0 | 0
c15e00f3c26d478b9de63617d296dee0 | 127.2.14.1:46525 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.2.14.1 | experimental | 127.2.14.1:46525
local_ip_for_outbound_sockets | 127.2.14.2 | experimental | 127.2.14.2:36527
local_ip_for_outbound_sockets | 127.2.14.3 | experimental | 127.2.14.3:40469
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data/info.pb | hidden | 127.2.14.1:46525
server_dump_info_path | /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data/info.pb | hidden | 127.2.14.2:36527
server_dump_info_path | /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data/info.pb | hidden | 127.2.14.3:40469
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.14.20:34391 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250624 14:21:09.876266 2104 log_verifier.cc:126] Checking tablet 37c156433d6d45af8a62d6f6b858c0ed
I20250624 14:21:10.182422 2104 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 37c156433d6d45af8a62d6f6b858c0ed
I20250624 14:21:10.185055 8467 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53660:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250624 14:21:10.185576 8467 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53660:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250624 14:21:10.200266 8467 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 6cdddb09ff5343ba92a6b6fd03bc4b75: Sending DeleteTablet for 1 replicas of tablet 37c156433d6d45af8a62d6f6b858c0ed
I20250624 14:21:10.202330 2104 test_util.cc:276] Using random seed: 171464643
I20250624 14:21:10.202663 8727 tablet_service.cc:1515] Processing DeleteTablet for tablet 37c156433d6d45af8a62d6f6b858c0ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-24 14:21:10 UTC) from {username='slave'} at 127.0.0.1:52252
I20250624 14:21:10.204609 9008 tablet_replica.cc:331] stopping tablet replica
I20250624 14:21:10.205530 9008 raft_consensus.cc:2241] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 2 LEADER]: Raft consensus shutting down.
I20250624 14:21:10.206413 9008 raft_consensus.cc:2270] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250624 14:21:10.237687 8467 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53696:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 14:21:10.241019 9008 ts_tablet_manager.cc:1905] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250624 14:21:10.241796 8467 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 14:21:10.254185 9008 ts_tablet_manager.cc:1918] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250624 14:21:10.254771 9008 log.cc:1199] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Deleting WAL directory at /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/wal/wals/37c156433d6d45af8a62d6f6b858c0ed
I20250624 14:21:10.255822 9008 ts_tablet_manager.cc:1939] T 37c156433d6d45af8a62d6f6b858c0ed P 45a4ffd4b59c43d997373faec35cc264: Deleting consensus metadata
I20250624 14:21:10.259275 8452 catalog_manager.cc:4928] TS 45a4ffd4b59c43d997373faec35cc264 (127.2.14.2:36527): tablet 37c156433d6d45af8a62d6f6b858c0ed (table pre_rebuild [id=f15ee8ca224543e1bc742c213bbd2ca1]) successfully deleted
I20250624 14:21:10.271332 8869 tablet_service.cc:1468] Processing CreateTablet for tablet d3b0990334f34475bc17df36f37386ad (DEFAULT_TABLE table=post_rebuild [id=ee452400a718488ba61843362a12c25e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:21:10.271431 8591 tablet_service.cc:1468] Processing CreateTablet for tablet d3b0990334f34475bc17df36f37386ad (DEFAULT_TABLE table=post_rebuild [id=ee452400a718488ba61843362a12c25e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:21:10.272938 8869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3b0990334f34475bc17df36f37386ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:21:10.273159 8591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3b0990334f34475bc17df36f37386ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:21:10.274971 8727 tablet_service.cc:1468] Processing CreateTablet for tablet d3b0990334f34475bc17df36f37386ad (DEFAULT_TABLE table=post_rebuild [id=ee452400a718488ba61843362a12c25e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 14:21:10.276443 8727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3b0990334f34475bc17df36f37386ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 14:21:10.301559 9017 tablet_bootstrap.cc:492] T d3b0990334f34475bc17df36f37386ad P b25f654e2f274423880aee74a30d5dab: Bootstrap starting.
I20250624 14:21:10.299746 9015 tablet_bootstrap.cc:492] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0: Bootstrap starting.
I20250624 14:21:10.304729 9016 tablet_bootstrap.cc:492] T d3b0990334f34475bc17df36f37386ad P 45a4ffd4b59c43d997373faec35cc264: Bootstrap starting.
I20250624 14:21:10.308529 9017 tablet_bootstrap.cc:654] T d3b0990334f34475bc17df36f37386ad P b25f654e2f274423880aee74a30d5dab: Neither blocks nor log segments found. Creating new log.
I20250624 14:21:10.312093 9015 tablet_bootstrap.cc:654] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0: Neither blocks nor log segments found. Creating new log.
I20250624 14:21:10.316653 9017 tablet_bootstrap.cc:492] T d3b0990334f34475bc17df36f37386ad P b25f654e2f274423880aee74a30d5dab: No bootstrap required, opened a new log
I20250624 14:21:10.318500 9017 ts_tablet_manager.cc:1397] T d3b0990334f34475bc17df36f37386ad P b25f654e2f274423880aee74a30d5dab: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.005s
I20250624 14:21:10.322165 9015 tablet_bootstrap.cc:492] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0: No bootstrap required, opened a new log
I20250624 14:21:10.322666 9015 ts_tablet_manager.cc:1397] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.004s
I20250624 14:21:10.322649 9017 raft_consensus.cc:357] T d3b0990334f34475bc17df36f37386ad P b25f654e2f274423880aee74a30d5dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } }
I20250624 14:21:10.323509 9017 raft_consensus.cc:383] T d3b0990334f34475bc17df36f37386ad P b25f654e2f274423880aee74a30d5dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:21:10.324414 9016 tablet_bootstrap.cc:654] T d3b0990334f34475bc17df36f37386ad P 45a4ffd4b59c43d997373faec35cc264: Neither blocks nor log segments found. Creating new log.
I20250624 14:21:10.324457 9017 raft_consensus.cc:738] T d3b0990334f34475bc17df36f37386ad P b25f654e2f274423880aee74a30d5dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25f654e2f274423880aee74a30d5dab, State: Initialized, Role: FOLLOWER
I20250624 14:21:10.325815 9017 consensus_queue.cc:260] T d3b0990334f34475bc17df36f37386ad P b25f654e2f274423880aee74a30d5dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } }
I20250624 14:21:10.329003 9015 raft_consensus.cc:357] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } }
I20250624 14:21:10.329936 9015 raft_consensus.cc:383] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:21:10.330264 9015 raft_consensus.cc:738] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15e00f3c26d478b9de63617d296dee0, State: Initialized, Role: FOLLOWER
I20250624 14:21:10.331040 9015 consensus_queue.cc:260] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } }
I20250624 14:21:10.336566 9017 ts_tablet_manager.cc:1428] T d3b0990334f34475bc17df36f37386ad P b25f654e2f274423880aee74a30d5dab: Time spent starting tablet: real 0.018s user 0.013s sys 0.000s
I20250624 14:21:10.337247 9016 tablet_bootstrap.cc:492] T d3b0990334f34475bc17df36f37386ad P 45a4ffd4b59c43d997373faec35cc264: No bootstrap required, opened a new log
I20250624 14:21:10.337759 9016 ts_tablet_manager.cc:1397] T d3b0990334f34475bc17df36f37386ad P 45a4ffd4b59c43d997373faec35cc264: Time spent bootstrapping tablet: real 0.033s user 0.018s sys 0.000s
I20250624 14:21:10.341188 9016 raft_consensus.cc:357] T d3b0990334f34475bc17df36f37386ad P 45a4ffd4b59c43d997373faec35cc264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } }
I20250624 14:21:10.341951 9016 raft_consensus.cc:383] T d3b0990334f34475bc17df36f37386ad P 45a4ffd4b59c43d997373faec35cc264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 14:21:10.342268 9016 raft_consensus.cc:738] T d3b0990334f34475bc17df36f37386ad P 45a4ffd4b59c43d997373faec35cc264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a4ffd4b59c43d997373faec35cc264, State: Initialized, Role: FOLLOWER
I20250624 14:21:10.342657 9015 ts_tablet_manager.cc:1428] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0: Time spent starting tablet: real 0.020s user 0.003s sys 0.007s
I20250624 14:21:10.342979 9016 consensus_queue.cc:260] T d3b0990334f34475bc17df36f37386ad P 45a4ffd4b59c43d997373faec35cc264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } }
I20250624 14:21:10.347517 9016 ts_tablet_manager.cc:1428] T d3b0990334f34475bc17df36f37386ad P 45a4ffd4b59c43d997373faec35cc264: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20250624 14:21:10.449501 9022 raft_consensus.cc:491] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 14:21:10.450068 9022 raft_consensus.cc:513] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } }
W20250624 14:21:10.452368 8657 tablet.cc:2378] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:21:10.453423 9022 leader_election.cc:290] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469), 45a4ffd4b59c43d997373faec35cc264 (127.2.14.2:36527)
I20250624 14:21:10.454421 8747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3b0990334f34475bc17df36f37386ad" candidate_uuid: "c15e00f3c26d478b9de63617d296dee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a4ffd4b59c43d997373faec35cc264" is_pre_election: true
I20250624 14:21:10.455068 8747 raft_consensus.cc:2466] T d3b0990334f34475bc17df36f37386ad P 45a4ffd4b59c43d997373faec35cc264 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c15e00f3c26d478b9de63617d296dee0 in term 0.
I20250624 14:21:10.456660 8542 leader_election.cc:304] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a4ffd4b59c43d997373faec35cc264, c15e00f3c26d478b9de63617d296dee0; no voters:
I20250624 14:21:10.457563 9022 raft_consensus.cc:2802] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 14:21:10.457932 9022 raft_consensus.cc:491] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 14:21:10.458307 9022 raft_consensus.cc:3058] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:21:10.465627 9022 raft_consensus.cc:513] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } }
I20250624 14:21:10.468853 8747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3b0990334f34475bc17df36f37386ad" candidate_uuid: "c15e00f3c26d478b9de63617d296dee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a4ffd4b59c43d997373faec35cc264"
I20250624 14:21:10.469400 8747 raft_consensus.cc:3058] T d3b0990334f34475bc17df36f37386ad P 45a4ffd4b59c43d997373faec35cc264 [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:21:10.470702 9022 leader_election.cc:290] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [CANDIDATE]: Term 1 election: Requested vote from peers b25f654e2f274423880aee74a30d5dab (127.2.14.3:40469), 45a4ffd4b59c43d997373faec35cc264 (127.2.14.2:36527)
I20250624 14:21:10.476382 8747 raft_consensus.cc:2466] T d3b0990334f34475bc17df36f37386ad P 45a4ffd4b59c43d997373faec35cc264 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c15e00f3c26d478b9de63617d296dee0 in term 1.
I20250624 14:21:10.477574 8542 leader_election.cc:304] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a4ffd4b59c43d997373faec35cc264, c15e00f3c26d478b9de63617d296dee0; no voters:
I20250624 14:21:10.478397 9022 raft_consensus.cc:2802] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 14:21:10.481070 9022 raft_consensus.cc:695] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [term 1 LEADER]: Becoming Leader. State: Replica: c15e00f3c26d478b9de63617d296dee0, State: Running, Role: LEADER
I20250624 14:21:10.482283 8889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3b0990334f34475bc17df36f37386ad" candidate_uuid: "c15e00f3c26d478b9de63617d296dee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25f654e2f274423880aee74a30d5dab" is_pre_election: true
I20250624 14:21:10.482457 8888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3b0990334f34475bc17df36f37386ad" candidate_uuid: "c15e00f3c26d478b9de63617d296dee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25f654e2f274423880aee74a30d5dab"
I20250624 14:21:10.483069 8889 raft_consensus.cc:2466] T d3b0990334f34475bc17df36f37386ad P b25f654e2f274423880aee74a30d5dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c15e00f3c26d478b9de63617d296dee0 in term 0.
I20250624 14:21:10.483150 9022 consensus_queue.cc:237] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } } peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } }
W20250624 14:21:10.493496 8935 tablet.cc:2378] T d3b0990334f34475bc17df36f37386ad P b25f654e2f274423880aee74a30d5dab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:21:10.493038 8467 catalog_manager.cc:5582] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c15e00f3c26d478b9de63617d296dee0 (127.2.14.1). New cstate: current_term: 1 leader_uuid: "c15e00f3c26d478b9de63617d296dee0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c15e00f3c26d478b9de63617d296dee0" member_type: VOTER last_known_addr { host: "127.2.14.1" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 } health_report { overall_health: UNKNOWN } } }
W20250624 14:21:10.574041 8793 tablet.cc:2378] T d3b0990334f34475bc17df36f37386ad P 45a4ffd4b59c43d997373faec35cc264: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 14:21:10.626431 8747 raft_consensus.cc:1273] T d3b0990334f34475bc17df36f37386ad P 45a4ffd4b59c43d997373faec35cc264 [term 1 FOLLOWER]: Refusing update from remote peer c15e00f3c26d478b9de63617d296dee0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 14:21:10.628934 8889 raft_consensus.cc:3058] T d3b0990334f34475bc17df36f37386ad P b25f654e2f274423880aee74a30d5dab [term 0 FOLLOWER]: Advancing to term 1
I20250624 14:21:10.632902 9026 consensus_queue.cc:1035] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a4ffd4b59c43d997373faec35cc264" member_type: VOTER last_known_addr { host: "127.2.14.2" port: 36527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250624 14:21:10.637214 8889 raft_consensus.cc:1273] T d3b0990334f34475bc17df36f37386ad P b25f654e2f274423880aee74a30d5dab [term 1 FOLLOWER]: Refusing update from remote peer c15e00f3c26d478b9de63617d296dee0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 14:21:10.638998 9026 consensus_queue.cc:1035] T d3b0990334f34475bc17df36f37386ad P c15e00f3c26d478b9de63617d296dee0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25f654e2f274423880aee74a30d5dab" member_type: VOTER last_known_addr { host: "127.2.14.3" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 14:21:10.676235 9034 mvcc.cc:204] Tried to move back new op lower bound from 7171173870067494912 to 7171173869509058560. Current Snapshot: MvccSnapshot[applied={T|T < 7171173870067494912}]
I20250624 14:21:10.705176 9035 mvcc.cc:204] Tried to move back new op lower bound from 7171173870094503936 to 7171173869509058560. Current Snapshot: MvccSnapshot[applied={T|T < 7171173870067494912}]
W20250624 14:21:11.130280 9031 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s user 0.004s sys 0.000s
I20250624 14:21:15.899492 8727 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 14:21:15.901216 8869 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250624 14:21:15.910020 8591 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
6cdddb09ff5343ba92a6b6fd03bc4b75 | 127.2.14.62:33331 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.14.20:34391 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
45a4ffd4b59c43d997373faec35cc264 | 127.2.14.2:36527 | HEALTHY | <none> | 0 | 0
b25f654e2f274423880aee74a30d5dab | 127.2.14.3:40469 | HEALTHY | <none> | 0 | 0
c15e00f3c26d478b9de63617d296dee0 | 127.2.14.1:46525 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.2.14.1 | experimental | 127.2.14.1:46525
local_ip_for_outbound_sockets | 127.2.14.2 | experimental | 127.2.14.2:36527
local_ip_for_outbound_sockets | 127.2.14.3 | experimental | 127.2.14.3:40469
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-0/data/info.pb | hidden | 127.2.14.1:46525
server_dump_info_path | /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-1/data/info.pb | hidden | 127.2.14.2:36527
server_dump_info_path | /tmp/dist-test-taskDpG0JD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750774691568470-2104-0/minicluster-data/ts-2/data/info.pb | hidden | 127.2.14.3:40469
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.14.20:34391 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250624 14:21:16.155159 2104 log_verifier.cc:126] Checking tablet 37c156433d6d45af8a62d6f6b858c0ed
I20250624 14:21:16.156267 2104 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 37c156433d6d45af8a62d6f6b858c0ed
I20250624 14:21:16.156595 2104 log_verifier.cc:126] Checking tablet d3b0990334f34475bc17df36f37386ad
I20250624 14:21:16.914498 2104 log_verifier.cc:177] Verified matching terms for 205 ops in tablet d3b0990334f34475bc17df36f37386ad
I20250624 14:21:16.941097 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 8505
I20250624 14:21:16.986114 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 8660
I20250624 14:21:17.030206 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 8796
I20250624 14:21:17.069608 2104 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskDpG0JD/build/tsan/bin/kudu with pid 8435
2025-06-24T14:21:17Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (36252 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (36252 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (185494 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST
I20250624 14:21:17.141716 2104 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 4 messages since previous log ~48 seconds ago