Diagnosed failure

TsTabletManagerITest.TestTableStats: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:784: Failure
Expected equality of these values:
  live_row_count
    Which is: 67
  table_info->GetMetrics()->live_row_count->value()
    Which is: 0
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:402: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:768: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
  Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:786: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 784, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
  Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:896: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20261120 03:03:36.371076   385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:36.399328   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:36.399921   385 tablet_replica.cc:333] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: stopping tablet replica
I20261120 03:03:36.400439   385 raft_consensus.cc:2440] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.400847   385 raft_consensus.cc:2469] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.403102   385 tablet_replica.cc:333] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: stopping tablet replica
W20261120 03:03:36.403208  2164 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
I20261120 03:03:36.403661   385 raft_consensus.cc:2440] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 6 LEADER]: Raft consensus shutting down.
I20261120 03:03:36.404613   385 raft_consensus.cc:2469] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 6 FOLLOWER]: Raft consensus is shut down!
W20261120 03:03:36.405910  2164 consensus_peers.cc:597] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c -> Peer 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485): Couldn't send request to peer 9ac1cc0b02fd4724b4680da6a7e2a875. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:36.406553   385 tablet_replica.cc:333] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: stopping tablet replica
I20261120 03:03:36.406987   385 raft_consensus.cc:2440] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.407342   385 raft_consensus.cc:2469] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Raft consensus is shut down!
W20261120 03:03:36.435228  2164 consensus_peers.cc:597] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c -> Peer 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485): Couldn't send request to peer 9ac1cc0b02fd4724b4680da6a7e2a875. Status: Network error: Client connection negotiation failed: client connection to 127.0.96.65:40485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:36.461351   385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:36.475200   385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:03:36.504944  2664 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:36.505401  2664 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:36.506110   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:36.506793   385 tablet_replica.cc:333] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: stopping tablet replica
I20261120 03:03:36.507069  2664 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:36.507611   385 raft_consensus.cc:2440] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.508332   385 raft_consensus.cc:2469] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.509191  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:36.510725   385 tablet_replica.cc:333] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: stopping tablet replica
I20261120 03:03:36.511359   385 raft_consensus.cc:2440] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.511859   385 raft_consensus.cc:2469] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Raft consensus is shut down!
W20261120 03:03:36.512672  2086 leader_election.cc:336] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485): Network error: Client connection negotiation failed: client connection to 127.0.96.65:40485: connect: Connection refused (error 111)
I20261120 03:03:36.513085  2086 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
W20261120 03:03:36.513708  2086 raft_consensus.cc:2902] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20261120 03:03:36.514537   385 tablet_replica.cc:333] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: stopping tablet replica
I20261120 03:03:36.515241   385 raft_consensus.cc:2440] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.515856   385 raft_consensus.cc:2469] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Raft consensus is shut down!
W20261120 03:03:36.537401  2165 consensus_peers.cc:597] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c -> Peer a157015bf640485fb805159465a9ab1b (127.0.96.66:36367): Couldn't send request to peer a157015bf640485fb805159465a9ab1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.96.66:36367: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:36.546162   385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:03:36.559314   385 tablet_server.cc:179] TabletServer@127.0.96.67:0 shutting down...
W20261120 03:03:36.574368  2165 consensus_peers.cc:597] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c -> Peer a157015bf640485fb805159465a9ab1b (127.0.96.66:36367): Couldn't send request to peer a157015bf640485fb805159465a9ab1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.96.66:36367: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:36.585305   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:36.585899   385 tablet_replica.cc:333] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: stopping tablet replica
I20261120 03:03:36.586660   385 raft_consensus.cc:2440] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 6 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.587358   385 raft_consensus.cc:2469] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 6 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.589263   385 tablet_replica.cc:333] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: stopping tablet replica
I20261120 03:03:36.589651   385 raft_consensus.cc:2440] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 LEADER]: Raft consensus shutting down.
I20261120 03:03:36.590348   385 raft_consensus.cc:2469] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.592000   385 tablet_replica.cc:333] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: stopping tablet replica
I20261120 03:03:36.592484   385 raft_consensus.cc:2440] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 LEADER]: Raft consensus shutting down.
I20261120 03:03:36.593086   385 raft_consensus.cc:2469] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.614674   385 tablet_server.cc:196] TabletServer@127.0.96.67:0 shutdown complete.
I20261120 03:03:36.626992   385 master.cc:585] Master@127.0.96.126:43725 shutting down...
I20261120 03:03:36.647610   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 4 LEADER]: Raft consensus shutting down.
I20261120 03:03:36.648514   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.648893   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: stopping tablet replica
I20261120 03:03:36.669073   385 master.cc:607] Master@127.0.96.126:43725 shutdown complete.
I20261120 03:03:36.685151   385 master.cc:585] Master@127.0.96.125:41159 shutting down...
I20261120 03:03:36.704452   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.704883   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.705148   385 tablet_replica.cc:333] stopping tablet replica
I20261120 03:03:36.740841   385 master.cc:607] Master@127.0.96.125:41159 shutdown complete.
I20261120 03:03:36.753082   385 master.cc:585] Master@127.0.96.124:46415 shutting down...
I20261120 03:03:36.774307   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.774768   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.775199   385 tablet_replica.cc:333] stopping tablet replica
I20261120 03:03:36.793802   385 master.cc:607] Master@127.0.96.124:46415 shutdown complete.
I20261120 03:03:36.816525   385 test_util.cc:182] -----------------------------------------------
I20261120 03:03:36.816721   385 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0

Full log

[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN      ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20261120 03:02:50.040824   385 thread.cc:642] rpc reactor (reactor) Time spent creating pthread: real 1.117s	user 0.418s	sys 0.699s
W20261120 03:02:50.041303   385 thread.cc:609] rpc reactor (reactor) Time spent starting thread: real 1.118s	user 0.418s	sys 0.699s
I20261120 03:02:50.046075   385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:41885
I20261120 03:02:50.047827   385 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 03:02:50.049088   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:50.062438   396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:50.065327   397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:50.152732   385 server_base.cc:1061] running on GCE node
W20261120 03:02:50.152817   399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:50.154024   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:50.154284   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:50.154474   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143770154451 us; error 0 us; skew 500 ppm
I20261120 03:02:50.155157   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:50.159634   385 webserver.cc:577] Webserver started at http://127.0.96.126:43295/ using document root <none> and password file <none>
I20261120 03:02:50.160435   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:50.160648   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:50.161087   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:50.170782   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "e21e1011c1784b58ad643adc3e231996"
format_stamp: "Formatted at 2026-11-20 03:02:50 on dist-test-slave-54pf"
I20261120 03:02:50.177853   385 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.004s
I20261120 03:02:50.182910   404 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:50.183894   385 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20261120 03:02:50.184273   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "e21e1011c1784b58ad643adc3e231996"
format_stamp: "Formatted at 2026-11-20 03:02:50 on dist-test-slave-54pf"
I20261120 03:02:50.184577   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:50.205601   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:02:50.206974   385 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 03:02:50.207396   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:50.280997   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:41885
I20261120 03:02:50.281056   455 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:41885 every 8 connection(s)
I20261120 03:02:50.287717   456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:50.309675   456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996: Bootstrap starting.
I20261120 03:02:50.316427   456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:50.318010   456 log.cc:826] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996: Log is configured to *not* fsync() on all Append() calls
I20261120 03:02:50.322595   456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996: No bootstrap required, opened a new log
I20261120 03:02:50.342831   456 raft_consensus.cc:366] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21e1011c1784b58ad643adc3e231996" member_type: VOTER }
I20261120 03:02:50.343411   456 raft_consensus.cc:392] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:02:50.343658   456 raft_consensus.cc:787] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e21e1011c1784b58ad643adc3e231996, State: Initialized, Role: FOLLOWER
I20261120 03:02:50.344659   456 consensus_queue.cc:261] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21e1011c1784b58ad643adc3e231996" member_type: VOTER }
I20261120 03:02:50.345237   456 raft_consensus.cc:407] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:02:50.345546   456 raft_consensus.cc:534] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:02:50.345932   456 raft_consensus.cc:3296] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:50.348452   456 raft_consensus.cc:556] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21e1011c1784b58ad643adc3e231996" member_type: VOTER }
I20261120 03:02:50.352891   456 leader_election.cc:304] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e21e1011c1784b58ad643adc3e231996; no voters: 
I20261120 03:02:50.353484   456 leader_election.cc:290] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:02:50.353668   459 raft_consensus.cc:3024] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:02:50.356720   459 raft_consensus.cc:744] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 1 LEADER]: Becoming Leader. State: Replica: e21e1011c1784b58ad643adc3e231996, State: Running, Role: LEADER
I20261120 03:02:50.357458   459 consensus_queue.cc:238] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21e1011c1784b58ad643adc3e231996" member_type: VOTER }
I20261120 03:02:50.358515   456 sys_catalog.cc:565] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:02:50.367194   460 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e21e1011c1784b58ad643adc3e231996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21e1011c1784b58ad643adc3e231996" member_type: VOTER } }
I20261120 03:02:50.369271   460 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:50.367074   461 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e21e1011c1784b58ad643adc3e231996. Latest consensus state: current_term: 1 leader_uuid: "e21e1011c1784b58ad643adc3e231996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21e1011c1784b58ad643adc3e231996" member_type: VOTER } }
I20261120 03:02:50.370153   461 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:50.373601   468 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:02:50.385291   468 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:02:50.387223   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:02:50.400343   468 catalog_manager.cc:1398] Generated new cluster ID: 126ebf1cbedf4513b5180b763d1ced4a
I20261120 03:02:50.400591   468 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:02:50.424769   468 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:02:50.426559   468 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:02:50.449887   468 catalog_manager.cc:6237] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996: Generated new TSK 0
I20261120 03:02:50.450917   468 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:02:50.517576   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:50.527235   478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:50.529088   479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:50.533397   385 server_base.cc:1061] running on GCE node
W20261120 03:02:50.533456   481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:50.534327   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:50.534543   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:50.534742   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143770534725 us; error 0 us; skew 500 ppm
I20261120 03:02:50.535307   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:50.539877   385 webserver.cc:577] Webserver started at http://127.0.96.65:39301/ using document root <none> and password file <none>
I20261120 03:02:50.540427   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:50.540663   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:50.540948   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:50.542151   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "5447aa6248eb48c29da5c94ead234e39"
format_stamp: "Formatted at 2026-11-20 03:02:50 on dist-test-slave-54pf"
I20261120 03:02:50.547582   385 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.002s
I20261120 03:02:50.552083   486 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:50.553009   385 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20261120 03:02:50.553297   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "5447aa6248eb48c29da5c94ead234e39"
format_stamp: "Formatted at 2026-11-20 03:02:50 on dist-test-slave-54pf"
I20261120 03:02:50.553609   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:50.577068   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:02:50.578270   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:50.581223   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:50.585417   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:50.585598   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:50.585877   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:50.586064   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:50.696205   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:38505
I20261120 03:02:50.696272   548 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:38505 every 8 connection(s)
I20261120 03:02:50.703512   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:50.715857   553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:50.725775   554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:50.728582   385 server_base.cc:1061] running on GCE node
W20261120 03:02:50.728572   556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:50.729579   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:50.729799   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:50.730012   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143770729977 us; error 0 us; skew 500 ppm
I20261120 03:02:50.730557   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:50.733397   385 webserver.cc:577] Webserver started at http://127.0.96.66:38963/ using document root <none> and password file <none>
I20261120 03:02:50.733947   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:50.734176   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:50.734490   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:50.735833   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "ecfa61e8132642ac916dded8a129376a"
format_stamp: "Formatted at 2026-11-20 03:02:50 on dist-test-slave-54pf"
I20261120 03:02:50.737684   549 heartbeater.cc:345] Connected to a master server at 127.0.96.126:41885
I20261120 03:02:50.738142   549 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:50.739223   549 heartbeater.cc:508] Master 127.0.96.126:41885 requested a full tablet report, sending...
I20261120 03:02:50.742740   385 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20261120 03:02:50.743119   421 ts_manager.cc:210] Registered new tserver with Master: 5447aa6248eb48c29da5c94ead234e39 (127.0.96.65:38505)
I20261120 03:02:50.746047   421 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58360
I20261120 03:02:50.747254   561 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:50.748175   385 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20261120 03:02:50.748461   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "ecfa61e8132642ac916dded8a129376a"
format_stamp: "Formatted at 2026-11-20 03:02:50 on dist-test-slave-54pf"
I20261120 03:02:50.748803   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:50.767460   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:02:50.768630   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:50.770954   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:50.774176   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:50.774343   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:50.774616   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:50.774792   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:50.850265   549 heartbeater.cc:500] Master 127.0.96.126:41885 was elected leader, sending a full tablet report...
I20261120 03:02:50.879045   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:35013
I20261120 03:02:50.879148   623 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:35013 every 8 connection(s)
I20261120 03:02:50.894590   624 heartbeater.cc:345] Connected to a master server at 127.0.96.126:41885
I20261120 03:02:50.894910   624 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:50.895560   624 heartbeater.cc:508] Master 127.0.96.126:41885 requested a full tablet report, sending...
I20261120 03:02:50.897472   421 ts_manager.cc:210] Registered new tserver with Master: ecfa61e8132642ac916dded8a129376a (127.0.96.66:35013)
I20261120 03:02:50.898782   385 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.015125361s
I20261120 03:02:50.899519   421 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58364
I20261120 03:02:50.927897   421 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58366:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 2
    seed: 0
  }
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20261120 03:02:50.945463   421 catalog_manager.cc:7244] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20261120 03:02:50.997280   589 tablet_service.cc:1495] Processing CreateTablet for tablet b8b8567da20c4ae1a2a6c155b887254b (DEFAULT_TABLE table=test-table [id=3790e4dc82734b1d9ff1d740a1467d50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261120 03:02:50.999182   588 tablet_service.cc:1495] Processing CreateTablet for tablet 62c70f83df9048fe93523ea6df2addc9 (DEFAULT_TABLE table=test-table [id=3790e4dc82734b1d9ff1d740a1467d50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261120 03:02:50.999801   589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b8567da20c4ae1a2a6c155b887254b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:51.000559   588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62c70f83df9048fe93523ea6df2addc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:51.000428   514 tablet_service.cc:1495] Processing CreateTablet for tablet b8b8567da20c4ae1a2a6c155b887254b (DEFAULT_TABLE table=test-table [id=3790e4dc82734b1d9ff1d740a1467d50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261120 03:02:51.001675   514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b8567da20c4ae1a2a6c155b887254b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:51.002096   624 heartbeater.cc:500] Master 127.0.96.126:41885 was elected leader, sending a full tablet report...
I20261120 03:02:51.003746   513 tablet_service.cc:1495] Processing CreateTablet for tablet 62c70f83df9048fe93523ea6df2addc9 (DEFAULT_TABLE table=test-table [id=3790e4dc82734b1d9ff1d740a1467d50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261120 03:02:51.004851   513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62c70f83df9048fe93523ea6df2addc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:51.034216   640 tablet_bootstrap.cc:492] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a: Bootstrap starting.
I20261120 03:02:51.041764   641 tablet_bootstrap.cc:492] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39: Bootstrap starting.
I20261120 03:02:51.042625   640 tablet_bootstrap.cc:654] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:51.048375   640 tablet_bootstrap.cc:492] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a: No bootstrap required, opened a new log
I20261120 03:02:51.048725   640 ts_tablet_manager.cc:1449] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a: Time spent bootstrapping tablet: real 0.015s	user 0.014s	sys 0.000s
I20261120 03:02:51.049633   641 tablet_bootstrap.cc:654] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:51.051049   640 raft_consensus.cc:366] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.051544   640 raft_consensus.cc:787] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Initialized, Role: FOLLOWER
I20261120 03:02:51.052090   640 consensus_queue.cc:261] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.054401   640 ts_tablet_manager.cc:1480] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20261120 03:02:51.055697   640 tablet_bootstrap.cc:492] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a: Bootstrap starting.
I20261120 03:02:51.059751   641 tablet_bootstrap.cc:492] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39: No bootstrap required, opened a new log
I20261120 03:02:51.060336   641 ts_tablet_manager.cc:1449] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39: Time spent bootstrapping tablet: real 0.019s	user 0.009s	sys 0.007s
I20261120 03:02:51.062566   640 tablet_bootstrap.cc:654] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:51.063140   641 raft_consensus.cc:366] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } }
I20261120 03:02:51.063854   641 raft_consensus.cc:787] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5447aa6248eb48c29da5c94ead234e39, State: Initialized, Role: FOLLOWER
I20261120 03:02:51.064563   641 consensus_queue.cc:261] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } }
I20261120 03:02:51.082293   641 ts_tablet_manager.cc:1480] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.008s
I20261120 03:02:51.083113   641 tablet_bootstrap.cc:492] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39: Bootstrap starting.
I20261120 03:02:51.084401   640 tablet_bootstrap.cc:492] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a: No bootstrap required, opened a new log
I20261120 03:02:51.084937   640 ts_tablet_manager.cc:1449] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a: Time spent bootstrapping tablet: real 0.029s	user 0.018s	sys 0.010s
I20261120 03:02:51.087936   640 raft_consensus.cc:366] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } }
I20261120 03:02:51.088629   640 raft_consensus.cc:787] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Initialized, Role: FOLLOWER
I20261120 03:02:51.089255   640 consensus_queue.cc:261] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } }
I20261120 03:02:51.090127   641 tablet_bootstrap.cc:654] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:51.092713   640 ts_tablet_manager.cc:1480] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.001s
I20261120 03:02:51.105151   641 tablet_bootstrap.cc:492] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39: No bootstrap required, opened a new log
I20261120 03:02:51.105708   641 ts_tablet_manager.cc:1449] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39: Time spent bootstrapping tablet: real 0.023s	user 0.011s	sys 0.005s
I20261120 03:02:51.108824   641 raft_consensus.cc:366] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.109308   641 raft_consensus.cc:787] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5447aa6248eb48c29da5c94ead234e39, State: Initialized, Role: FOLLOWER
I20261120 03:02:51.109731   641 consensus_queue.cc:261] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.111485   641 ts_tablet_manager.cc:1480] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20261120 03:02:51.126739   385 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20261120 03:02:51.127024   385 raft_consensus.cc:433] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Emulating election...
I20261120 03:02:51.127271   385 raft_consensus.cc:3296] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:51.129714   385 raft_consensus.cc:744] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 1 LEADER]: Becoming Leader. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Running, Role: LEADER
I20261120 03:02:51.130369   385 consensus_queue.cc:238] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } }
W20261120 03:02:51.136704   625 tablet.cc:2554] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261120 03:02:51.138794   385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:51.145825   421 catalog_manager.cc:5856] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a reported cstate change: term changed from 0 to 1, leader changed from <none> to ecfa61e8132642ac916dded8a129376a (127.0.96.66). New cstate: current_term: 1 leader_uuid: "ecfa61e8132642ac916dded8a129376a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } health_report { overall_health: HEALTHY } } }
I20261120 03:02:51.155512   524 raft_consensus.cc:3296] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:51.164139   643 consensus_queue.cc:1049] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261120 03:02:51.171869   385 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20261120 03:02:51.276032   385 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20261120 03:02:51.276361   385 raft_consensus.cc:433] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Emulating election...
I20261120 03:02:51.276646   385 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:51.277360   385 raft_consensus.cc:744] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 1 LEADER]: Becoming Leader. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Running, Role: LEADER
I20261120 03:02:51.278007   385 consensus_queue.cc:238] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.282943   385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:51.284026   421 catalog_manager.cc:5856] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a reported cstate change: term changed from 0 to 1, leader changed from <none> to ecfa61e8132642ac916dded8a129376a (127.0.96.66). New cstate: current_term: 1 leader_uuid: "ecfa61e8132642ac916dded8a129376a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20261120 03:02:51.285388   523 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:51.295140   643 consensus_queue.cc:1049] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:02:51.297107   385 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20261120 03:02:51.601078   385 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20261120 03:02:51.601366   385 raft_consensus.cc:433] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 1 LEADER]: Emulating election...
I20261120 03:02:51.601702   385 raft_consensus.cc:3291] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 1 LEADER]: Stepping down as leader of term 1
I20261120 03:02:51.602078   385 raft_consensus.cc:787] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Running, Role: LEADER
I20261120 03:02:51.602891   385 consensus_queue.cc:261] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.603915   385 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:02:51.604655   385 raft_consensus.cc:744] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 2 LEADER]: Becoming Leader. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Running, Role: LEADER
I20261120 03:02:51.605245   385 consensus_queue.cc:238] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.609562   385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:51.610901   420 catalog_manager.cc:5856] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "ecfa61e8132642ac916dded8a129376a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20261120 03:02:51.611577   523 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:02:51.612534   523 raft_consensus.cc:1338] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 2 FOLLOWER]: Refusing update from remote peer ecfa61e8132642ac916dded8a129376a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20261120 03:02:51.614773   385 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20261120 03:02:51.619625   650 consensus_queue.cc:1049] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261120 03:02:51.718739   385 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20261120 03:02:51.719038   385 raft_consensus.cc:433] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 2 FOLLOWER]: Emulating election...
I20261120 03:02:51.719286   385 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:02:51.721539   385 raft_consensus.cc:744] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 3 LEADER]: Becoming Leader. State: Replica: 5447aa6248eb48c29da5c94ead234e39, State: Running, Role: LEADER
I20261120 03:02:51.722182   385 consensus_queue.cc:238] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.726184   385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:51.727814   420 catalog_manager.cc:5856] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 reported cstate change: term changed from 2 to 3, leader changed from ecfa61e8132642ac916dded8a129376a (127.0.96.66) to 5447aa6248eb48c29da5c94ead234e39 (127.0.96.65). New cstate: current_term: 3 leader_uuid: "5447aa6248eb48c29da5c94ead234e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } health_report { overall_health: HEALTHY } } }
I20261120 03:02:51.732304   385 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20261120 03:02:51.752517   599 raft_consensus.cc:3291] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 2 LEADER]: Stepping down as leader of term 2
I20261120 03:02:51.752867   599 raft_consensus.cc:787] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Running, Role: LEADER
I20261120 03:02:51.753482   599 consensus_queue.cc:261] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:51.754279   599 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:02:51.755033   599 raft_consensus.cc:1338] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 3 FOLLOWER]: Refusing update from remote peer 5447aa6248eb48c29da5c94ead234e39: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20261120 03:02:51.759758   646 consensus_queue.cc:1049] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:02:52.036280   385 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20261120 03:02:52.036523   385 raft_consensus.cc:433] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 1 FOLLOWER]: Emulating election...
I20261120 03:02:52.036758   385 raft_consensus.cc:3296] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:02:52.037344   385 raft_consensus.cc:744] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 2 LEADER]: Becoming Leader. State: Replica: 5447aa6248eb48c29da5c94ead234e39, State: Running, Role: LEADER
I20261120 03:02:52.037969   385 consensus_queue.cc:238] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } }
I20261120 03:02:52.042675   419 catalog_manager.cc:5856] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 reported cstate change: term changed from 1 to 2, leader changed from ecfa61e8132642ac916dded8a129376a (127.0.96.66) to 5447aa6248eb48c29da5c94ead234e39 (127.0.96.65). New cstate: current_term: 2 leader_uuid: "5447aa6248eb48c29da5c94ead234e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } health_report { overall_health: UNKNOWN } } }
I20261120 03:02:52.044062   385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:52.046386   599 raft_consensus.cc:3291] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 1 LEADER]: Stepping down as leader of term 1
I20261120 03:02:52.046701   599 raft_consensus.cc:787] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Running, Role: LEADER
I20261120 03:02:52.047257   599 consensus_queue.cc:261] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } } peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } }
I20261120 03:02:52.048110   599 raft_consensus.cc:3296] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:02:52.052510   599 raft_consensus.cc:1338] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 2 FOLLOWER]: Refusing update from remote peer 5447aa6248eb48c29da5c94ead234e39: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20261120 03:02:52.055073   385 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20261120 03:02:52.055882   659 consensus_queue.cc:1049] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261120 03:02:52.159601   385 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20261120 03:02:52.159861   385 raft_consensus.cc:433] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 3 FOLLOWER]: Emulating election...
I20261120 03:02:52.160163   385 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:02:52.161406   385 raft_consensus.cc:744] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 4 LEADER]: Becoming Leader. State: Replica: ecfa61e8132642ac916dded8a129376a, State: Running, Role: LEADER
I20261120 03:02:52.162057   385 consensus_queue.cc:238] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:52.167339   385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:52.166570   420 catalog_manager.cc:5856] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a reported cstate change: term changed from 3 to 4, leader changed from 5447aa6248eb48c29da5c94ead234e39 (127.0.96.65) to ecfa61e8132642ac916dded8a129376a (127.0.96.66). New cstate: current_term: 4 leader_uuid: "ecfa61e8132642ac916dded8a129376a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } health_report { overall_health: UNKNOWN } } }
I20261120 03:02:52.169602   524 raft_consensus.cc:3291] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 3 LEADER]: Stepping down as leader of term 3
I20261120 03:02:52.169952   524 raft_consensus.cc:787] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5447aa6248eb48c29da5c94ead234e39, State: Running, Role: LEADER
I20261120 03:02:52.170550   524 consensus_queue.cc:261] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfa61e8132642ac916dded8a129376a" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 35013 } } peers { permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 } }
I20261120 03:02:52.171424   524 raft_consensus.cc:3296] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:02:52.172834   524 raft_consensus.cc:1338] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 4 FOLLOWER]: Refusing update from remote peer ecfa61e8132642ac916dded8a129376a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20261120 03:02:52.185900   643 consensus_queue.cc:1049] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5447aa6248eb48c29da5c94ead234e39" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:02:52.188866   385 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20261120 03:02:52.494252   385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:02:52.514456   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:02:52.515002   385 tablet_replica.cc:333] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39: stopping tablet replica
I20261120 03:02:52.515501   385 raft_consensus.cc:2440] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 4 FOLLOWER]: Raft consensus shutting down.
I20261120 03:02:52.515935   385 raft_consensus.cc:2469] T b8b8567da20c4ae1a2a6c155b887254b P 5447aa6248eb48c29da5c94ead234e39 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:52.517999   385 tablet_replica.cc:333] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39: stopping tablet replica
I20261120 03:02:52.518399   385 raft_consensus.cc:2440] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 2 LEADER]: Raft consensus shutting down.
I20261120 03:02:52.518859   385 raft_consensus.cc:2469] T 62c70f83df9048fe93523ea6df2addc9 P 5447aa6248eb48c29da5c94ead234e39 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:52.595891   385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:02:52.604504   385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:02:52.620702   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:02:52.621222   385 tablet_replica.cc:333] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a: stopping tablet replica
I20261120 03:02:52.621672   385 raft_consensus.cc:2440] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 2 FOLLOWER]: Raft consensus shutting down.
I20261120 03:02:52.622051   385 raft_consensus.cc:2469] T 62c70f83df9048fe93523ea6df2addc9 P ecfa61e8132642ac916dded8a129376a [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:52.623639   385 tablet_replica.cc:333] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a: stopping tablet replica
I20261120 03:02:52.624065   385 raft_consensus.cc:2440] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 4 LEADER]: Raft consensus shutting down.
I20261120 03:02:52.624585   385 raft_consensus.cc:2469] T b8b8567da20c4ae1a2a6c155b887254b P ecfa61e8132642ac916dded8a129376a [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:52.632028   385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:02:52.639272   385 master.cc:585] Master@127.0.96.126:41885 shutting down...
I20261120 03:02:52.651302   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:02:52.651695   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:52.652007   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P e21e1011c1784b58ad643adc3e231996: stopping tablet replica
I20261120 03:02:52.670928   385 master.cc:607] Master@127.0.96.126:41885 shutdown complete.
[       OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (3817 ms)
[ RUN      ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20261120 03:02:52.703197   385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:41183
I20261120 03:02:52.704322   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:52.708534   669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:52.709473   670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:52.712595   385 server_base.cc:1061] running on GCE node
W20261120 03:02:52.713920   672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:52.714552   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:52.714717   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:52.714823   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143772714812 us; error 0 us; skew 500 ppm
I20261120 03:02:52.715188   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:52.717239   385 webserver.cc:577] Webserver started at http://127.0.96.126:33509/ using document root <none> and password file <none>
I20261120 03:02:52.717592   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:52.717722   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:52.717895   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:52.718743   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "d085122c19f84ee8b64d592e736fe55e"
format_stamp: "Formatted at 2026-11-20 03:02:52 on dist-test-slave-54pf"
I20261120 03:02:52.722698   385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20261120 03:02:52.725286   677 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:52.725940   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20261120 03:02:52.726212   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "d085122c19f84ee8b64d592e736fe55e"
format_stamp: "Formatted at 2026-11-20 03:02:52 on dist-test-slave-54pf"
I20261120 03:02:52.726452   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:52.738395   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:02:52.739239   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:52.770321   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:41183
I20261120 03:02:52.770411   728 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:41183 every 8 connection(s)
I20261120 03:02:52.773727   729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:52.782223   729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e: Bootstrap starting.
I20261120 03:02:52.785889   729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:52.789340   729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e: No bootstrap required, opened a new log
I20261120 03:02:52.790975   729 raft_consensus.cc:366] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085122c19f84ee8b64d592e736fe55e" member_type: VOTER }
I20261120 03:02:52.791342   729 raft_consensus.cc:392] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:02:52.791579   729 raft_consensus.cc:787] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d085122c19f84ee8b64d592e736fe55e, State: Initialized, Role: FOLLOWER
I20261120 03:02:52.792095   729 consensus_queue.cc:261] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085122c19f84ee8b64d592e736fe55e" member_type: VOTER }
I20261120 03:02:52.792557   729 raft_consensus.cc:407] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:02:52.792801   729 raft_consensus.cc:534] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:02:52.793094   729 raft_consensus.cc:3296] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:52.794075   729 raft_consensus.cc:556] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085122c19f84ee8b64d592e736fe55e" member_type: VOTER }
I20261120 03:02:52.797783   729 leader_election.cc:304] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d085122c19f84ee8b64d592e736fe55e; no voters: 
I20261120 03:02:52.798237   729 leader_election.cc:290] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:02:52.798336   732 raft_consensus.cc:3024] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:02:52.799775   732 raft_consensus.cc:744] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 1 LEADER]: Becoming Leader. State: Replica: d085122c19f84ee8b64d592e736fe55e, State: Running, Role: LEADER
I20261120 03:02:52.800534   732 consensus_queue.cc:238] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085122c19f84ee8b64d592e736fe55e" member_type: VOTER }
I20261120 03:02:52.801303   729 sys_catalog.cc:565] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:02:52.803035   733 sys_catalog.cc:455] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d085122c19f84ee8b64d592e736fe55e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085122c19f84ee8b64d592e736fe55e" member_type: VOTER } }
I20261120 03:02:52.803068   734 sys_catalog.cc:455] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d085122c19f84ee8b64d592e736fe55e. Latest consensus state: current_term: 1 leader_uuid: "d085122c19f84ee8b64d592e736fe55e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d085122c19f84ee8b64d592e736fe55e" member_type: VOTER } }
I20261120 03:02:52.803674   734 sys_catalog.cc:458] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:52.803710   733 sys_catalog.cc:458] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:52.806228   737 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:02:52.810657   737 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:02:52.815560   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:02:52.818178   737 catalog_manager.cc:1398] Generated new cluster ID: 3c40363c2d3043b081be9496e3d3f8af
I20261120 03:02:52.818419   737 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:02:52.833510   737 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:02:52.834632   737 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:02:52.848431   737 catalog_manager.cc:6237] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e: Generated new TSK 0
I20261120 03:02:52.848946   737 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:02:52.881249   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:52.885972   750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:52.887203   751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:52.888425   753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:52.889441   385 server_base.cc:1061] running on GCE node
I20261120 03:02:52.890195   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:52.890373   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:52.890478   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143772890468 us; error 0 us; skew 500 ppm
I20261120 03:02:52.890854   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:52.892714   385 webserver.cc:577] Webserver started at http://127.0.96.65:36055/ using document root <none> and password file <none>
I20261120 03:02:52.893085   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:52.893235   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:52.893430   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:52.894299   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "0841085fdf594225b62f17e075964efc"
format_stamp: "Formatted at 2026-11-20 03:02:52 on dist-test-slave-54pf"
I20261120 03:02:52.897943   385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20261120 03:02:52.900786   758 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:52.901484   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20261120 03:02:52.901763   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "0841085fdf594225b62f17e075964efc"
format_stamp: "Formatted at 2026-11-20 03:02:52 on dist-test-slave-54pf"
I20261120 03:02:52.902033   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:52.921432   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:02:52.922277   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:52.923404   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:52.925359   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:52.925571   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:52.925797   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:52.925963   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:52.958354   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:35779
I20261120 03:02:52.958458   820 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:35779 every 8 connection(s)
I20261120 03:02:52.962819   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:52.969135   825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:52.969609   826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:52.973672   828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:52.974501   385 server_base.cc:1061] running on GCE node
I20261120 03:02:52.975126   821 heartbeater.cc:345] Connected to a master server at 127.0.96.126:41183
I20261120 03:02:52.975299   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20261120 03:02:52.975476   821 heartbeater.cc:462] Registering TS with master...
W20261120 03:02:52.975586   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:52.975855   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143772975838 us; error 0 us; skew 500 ppm
I20261120 03:02:52.976331   821 heartbeater.cc:508] Master 127.0.96.126:41183 requested a full tablet report, sending...
I20261120 03:02:52.976385   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:52.978284   694 ts_manager.cc:210] Registered new tserver with Master: 0841085fdf594225b62f17e075964efc (127.0.96.65:35779)
I20261120 03:02:52.978859   385 webserver.cc:577] Webserver started at http://127.0.96.66:33631/ using document root <none> and password file <none>
I20261120 03:02:52.979329   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:52.979532   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:52.979789   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:52.980060   694 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51876
I20261120 03:02:52.980968   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "afdf2aa455bd4fcdbd94920f8c83c062"
format_stamp: "Formatted at 2026-11-20 03:02:52 on dist-test-slave-54pf"
I20261120 03:02:52.984881   385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.000s	sys 0.004s
I20261120 03:02:52.987715   833 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:52.988490   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.002s
I20261120 03:02:52.988757   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "afdf2aa455bd4fcdbd94920f8c83c062"
format_stamp: "Formatted at 2026-11-20 03:02:52 on dist-test-slave-54pf"
I20261120 03:02:52.989024   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:53.006491   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:02:53.007314   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:53.008454   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:53.010470   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:53.010668   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:53.010877   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:53.011027   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:53.044147   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:41527
I20261120 03:02:53.044269   895 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:41527 every 8 connection(s)
I20261120 03:02:53.056586   896 heartbeater.cc:345] Connected to a master server at 127.0.96.126:41183
I20261120 03:02:53.056922   896 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:53.057722   896 heartbeater.cc:508] Master 127.0.96.126:41183 requested a full tablet report, sending...
I20261120 03:02:53.059435   694 ts_manager.cc:210] Registered new tserver with Master: afdf2aa455bd4fcdbd94920f8c83c062 (127.0.96.66:41527)
I20261120 03:02:53.060256   385 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.013017822s
I20261120 03:02:53.060889   694 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51878
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:422: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20261120 03:02:53.079242   385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:02:53.082311   821 heartbeater.cc:500] Master 127.0.96.126:41183 was elected leader, sending a full tablet report...
I20261120 03:02:53.094622   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:02:53.109112   385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:02:53.115116   385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:02:53.127990   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:02:53.142519   385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:02:53.148401   385 master.cc:585] Master@127.0.96.126:41183 shutting down...
I20261120 03:02:53.218262   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:02:53.218716   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:53.219066   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P d085122c19f84ee8b64d592e736fe55e: stopping tablet replica
I20261120 03:02:53.227041   385 master.cc:607] Master@127.0.96.126:41183 shutdown complete.
[  SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (546 ms)
[ RUN      ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20261120 03:02:53.247874   385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:43817
I20261120 03:02:53.248777   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:53.252965   909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:53.254076   910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:53.255172   912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:53.255730   385 server_base.cc:1061] running on GCE node
I20261120 03:02:53.256554   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:53.256721   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:53.256832   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143773256822 us; error 0 us; skew 500 ppm
I20261120 03:02:53.257277   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:53.259178   385 webserver.cc:577] Webserver started at http://127.0.96.126:46515/ using document root <none> and password file <none>
I20261120 03:02:53.259598   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:53.259807   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:53.259994   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:53.261003   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "88e7003c263548a1a47d5399f68474db"
format_stamp: "Formatted at 2026-11-20 03:02:53 on dist-test-slave-54pf"
I20261120 03:02:53.264917   385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.000s	sys 0.004s
I20261120 03:02:53.267578   917 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:53.268329   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20261120 03:02:53.268589   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "88e7003c263548a1a47d5399f68474db"
format_stamp: "Formatted at 2026-11-20 03:02:53 on dist-test-slave-54pf"
I20261120 03:02:53.268865   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:53.293268   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:02:53.294150   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:53.325850   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:43817
I20261120 03:02:53.325927   968 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:43817 every 8 connection(s)
I20261120 03:02:53.329370   969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:53.337953   969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db: Bootstrap starting.
I20261120 03:02:53.341802   969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:53.344969   969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db: No bootstrap required, opened a new log
I20261120 03:02:53.346736   969 raft_consensus.cc:366] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e7003c263548a1a47d5399f68474db" member_type: VOTER }
I20261120 03:02:53.347084   969 raft_consensus.cc:392] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:02:53.347312   969 raft_consensus.cc:787] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e7003c263548a1a47d5399f68474db, State: Initialized, Role: FOLLOWER
I20261120 03:02:53.347862   969 consensus_queue.cc:261] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e7003c263548a1a47d5399f68474db" member_type: VOTER }
I20261120 03:02:53.348318   969 raft_consensus.cc:407] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:02:53.348536   969 raft_consensus.cc:534] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:02:53.348821   969 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:53.349818   969 raft_consensus.cc:556] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e7003c263548a1a47d5399f68474db" member_type: VOTER }
I20261120 03:02:53.353677   969 leader_election.cc:304] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e7003c263548a1a47d5399f68474db; no voters: 
I20261120 03:02:53.354095   969 leader_election.cc:290] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:02:53.354226   972 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:02:53.356175   972 raft_consensus.cc:744] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 1 LEADER]: Becoming Leader. State: Replica: 88e7003c263548a1a47d5399f68474db, State: Running, Role: LEADER
I20261120 03:02:53.356801   972 consensus_queue.cc:238] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e7003c263548a1a47d5399f68474db" member_type: VOTER }
I20261120 03:02:53.357623   969 sys_catalog.cc:565] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:02:53.359772   974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88e7003c263548a1a47d5399f68474db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e7003c263548a1a47d5399f68474db" member_type: VOTER } }
I20261120 03:02:53.359845   973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e7003c263548a1a47d5399f68474db. Latest consensus state: current_term: 1 leader_uuid: "88e7003c263548a1a47d5399f68474db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e7003c263548a1a47d5399f68474db" member_type: VOTER } }
I20261120 03:02:53.360507   973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:53.360508   974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:53.363005   977 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:02:53.367304   977 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:02:53.373494   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:02:53.375591   977 catalog_manager.cc:1398] Generated new cluster ID: af685986090843a89a7f9861a371444d
I20261120 03:02:53.375824   977 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:02:53.398687   977 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:02:53.399760   977 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:02:53.413237   977 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db: Generated new TSK 0
I20261120 03:02:53.413787   977 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:02:53.439743   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:53.444774   990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:53.445981   991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:53.447175   993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:53.447427   385 server_base.cc:1061] running on GCE node
I20261120 03:02:53.448102   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:53.448333   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:53.448442   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143773448432 us; error 0 us; skew 500 ppm
I20261120 03:02:53.448880   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:53.450809   385 webserver.cc:577] Webserver started at http://127.0.96.65:45485/ using document root <none> and password file <none>
I20261120 03:02:53.451195   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:53.451337   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:53.451522   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:53.452531   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "a4e1c705ec064e7eb74a40c9a99fd34c"
format_stamp: "Formatted at 2026-11-20 03:02:53 on dist-test-slave-54pf"
I20261120 03:02:53.456312   385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20261120 03:02:53.459008   998 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:53.459815   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20261120 03:02:53.460036   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "a4e1c705ec064e7eb74a40c9a99fd34c"
format_stamp: "Formatted at 2026-11-20 03:02:53 on dist-test-slave-54pf"
I20261120 03:02:53.460351   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:53.489024   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:02:53.489845   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:53.491003   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:53.492938   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:53.493147   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:53.493391   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:53.493544   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:53.527350   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:36709
I20261120 03:02:53.527491  1060 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:36709 every 8 connection(s)
I20261120 03:02:53.531857   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:53.538072  1065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:53.538353  1066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:53.542657  1068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:53.543097   385 server_base.cc:1061] running on GCE node
I20261120 03:02:53.543116  1061 heartbeater.cc:345] Connected to a master server at 127.0.96.126:43817
I20261120 03:02:53.543638  1061 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:53.544013   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:53.544279   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:53.544404  1061 heartbeater.cc:508] Master 127.0.96.126:43817 requested a full tablet report, sending...
I20261120 03:02:53.544466   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143773544447 us; error 0 us; skew 500 ppm
I20261120 03:02:53.545184   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:53.546520   934 ts_manager.cc:210] Registered new tserver with Master: a4e1c705ec064e7eb74a40c9a99fd34c (127.0.96.65:36709)
I20261120 03:02:53.547653   385 webserver.cc:577] Webserver started at http://127.0.96.66:36101/ using document root <none> and password file <none>
I20261120 03:02:53.548156   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:53.548355   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:53.548326   934 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36312
I20261120 03:02:53.548715   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:53.549844   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "d2f8f09810044404b290ce30d0f28d4b"
format_stamp: "Formatted at 2026-11-20 03:02:53 on dist-test-slave-54pf"
I20261120 03:02:53.553897   385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20261120 03:02:53.556492  1073 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:53.557238   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I20261120 03:02:53.557489   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "d2f8f09810044404b290ce30d0f28d4b"
format_stamp: "Formatted at 2026-11-20 03:02:53 on dist-test-slave-54pf"
I20261120 03:02:53.557770   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:53.575754   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:02:53.576598   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:53.577756   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:53.579698   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:53.579913   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:53.580224   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:53.580415   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:53.613437   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:46787
I20261120 03:02:53.613518  1135 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:46787 every 8 connection(s)
I20261120 03:02:53.625835  1136 heartbeater.cc:345] Connected to a master server at 127.0.96.126:43817
I20261120 03:02:53.626174  1136 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:53.626785  1136 heartbeater.cc:508] Master 127.0.96.126:43817 requested a full tablet report, sending...
I20261120 03:02:53.628545   934 ts_manager.cc:210] Registered new tserver with Master: d2f8f09810044404b290ce30d0f28d4b (127.0.96.66:46787)
I20261120 03:02:53.629223   385 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.013028441s
I20261120 03:02:53.629799   934 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36314
I20261120 03:02:53.648029   934 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36320:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20261120 03:02:53.649295   934 catalog_manager.cc:7244] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20261120 03:02:53.650835  1061 heartbeater.cc:500] Master 127.0.96.126:43817 was elected leader, sending a full tablet report...
I20261120 03:02:53.679777  1026 tablet_service.cc:1495] Processing CreateTablet for tablet 43123be7b9fc4ec2b0e23685d1605874 (DEFAULT_TABLE table=test-table [id=6e54636ea85345349db164550c1c7667]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:02:53.680243  1101 tablet_service.cc:1495] Processing CreateTablet for tablet 43123be7b9fc4ec2b0e23685d1605874 (DEFAULT_TABLE table=test-table [id=6e54636ea85345349db164550c1c7667]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:02:53.680871  1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43123be7b9fc4ec2b0e23685d1605874. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:53.681277  1101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43123be7b9fc4ec2b0e23685d1605874. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:53.691589  1149 tablet_bootstrap.cc:492] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c: Bootstrap starting.
I20261120 03:02:53.692672  1150 tablet_bootstrap.cc:492] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b: Bootstrap starting.
I20261120 03:02:53.696022  1149 tablet_bootstrap.cc:654] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:53.696910  1150 tablet_bootstrap.cc:654] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:53.700207  1149 tablet_bootstrap.cc:492] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c: No bootstrap required, opened a new log
I20261120 03:02:53.700520  1150 tablet_bootstrap.cc:492] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b: No bootstrap required, opened a new log
I20261120 03:02:53.700572  1149 ts_tablet_manager.cc:1449] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c: Time spent bootstrapping tablet: real 0.009s	user 0.008s	sys 0.000s
I20261120 03:02:53.700865  1150 ts_tablet_manager.cc:1449] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.003s
I20261120 03:02:53.702297  1149 raft_consensus.cc:366] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.702448  1150 raft_consensus.cc:366] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.702750  1149 raft_consensus.cc:787] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e1c705ec064e7eb74a40c9a99fd34c, State: Initialized, Role: FOLLOWER
I20261120 03:02:53.702919  1150 raft_consensus.cc:787] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f8f09810044404b290ce30d0f28d4b, State: Initialized, Role: FOLLOWER
I20261120 03:02:53.703248  1149 consensus_queue.cc:261] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.703392  1150 consensus_queue.cc:261] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.705753  1149 ts_tablet_manager.cc:1480] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20261120 03:02:53.706166  1136 heartbeater.cc:500] Master 127.0.96.126:43817 was elected leader, sending a full tablet report...
I20261120 03:02:53.706687  1150 ts_tablet_manager.cc:1480] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20261120 03:02:53.734054   385 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20261120 03:02:53.734243   385 raft_consensus.cc:433] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 0 FOLLOWER]: Emulating election...
I20261120 03:02:53.734408   385 raft_consensus.cc:3296] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:53.735525   385 raft_consensus.cc:744] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 1 LEADER]: Becoming Leader. State: Replica: a4e1c705ec064e7eb74a40c9a99fd34c, State: Running, Role: LEADER
W20261120 03:02:53.736013  1061 heartbeater.cc:648] Failed to heartbeat to 127.0.96.126:43817 (0 consecutive failures): IO error: failing all heartbeats for tests
I20261120 03:02:53.736195   385 consensus_queue.cc:238] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.740449   385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:53.750219  1111 raft_consensus.cc:3296] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:53.757679  1156 consensus_queue.cc:1049] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:02:53.760089   385 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0
I20261120 03:02:53.865373   385 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20261120 03:02:53.865636   385 raft_consensus.cc:433] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 1 LEADER]: Emulating election...
I20261120 03:02:53.865844   385 raft_consensus.cc:3291] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 1 LEADER]: Stepping down as leader of term 1
I20261120 03:02:53.866024   385 raft_consensus.cc:787] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4e1c705ec064e7eb74a40c9a99fd34c, State: Running, Role: LEADER
I20261120 03:02:53.866446   385 consensus_queue.cc:261] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.867033   385 raft_consensus.cc:3296] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:02:53.867514   385 raft_consensus.cc:744] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 2 LEADER]: Becoming Leader. State: Replica: a4e1c705ec064e7eb74a40c9a99fd34c, State: Running, Role: LEADER
I20261120 03:02:53.868039   385 consensus_queue.cc:238] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.872304   385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:53.873689  1111 raft_consensus.cc:3296] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:02:53.874851  1111 raft_consensus.cc:1338] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 2 FOLLOWER]: Refusing update from remote peer a4e1c705ec064e7eb74a40c9a99fd34c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20261120 03:02:53.875900   385 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20261120 03:02:53.879434  1156 consensus_queue.cc:1049] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261120 03:02:53.980484   385 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20261120 03:02:53.980762   385 raft_consensus.cc:433] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 2 LEADER]: Emulating election...
I20261120 03:02:53.980983   385 raft_consensus.cc:3291] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 2 LEADER]: Stepping down as leader of term 2
I20261120 03:02:53.981207   385 raft_consensus.cc:787] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a4e1c705ec064e7eb74a40c9a99fd34c, State: Running, Role: LEADER
I20261120 03:02:53.981604   385 consensus_queue.cc:261] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.982203   385 raft_consensus.cc:3296] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:02:53.982753   385 raft_consensus.cc:744] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 3 LEADER]: Becoming Leader. State: Replica: a4e1c705ec064e7eb74a40c9a99fd34c, State: Running, Role: LEADER
I20261120 03:02:53.983289   385 consensus_queue.cc:238] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e1c705ec064e7eb74a40c9a99fd34c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36709 } } peers { permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 } }
I20261120 03:02:53.986923   385 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20261120 03:02:53.988339  1111 raft_consensus.cc:3296] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:02:53.989166  1111 raft_consensus.cc:1338] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 3 FOLLOWER]: Refusing update from remote peer a4e1c705ec064e7eb74a40c9a99fd34c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20261120 03:02:53.990523   385 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20261120 03:02:53.994103  1161 consensus_queue.cc:1049] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2f8f09810044404b290ce30d0f28d4b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:02:54.097016   385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:02:54.113597   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:02:54.114262   385 tablet_replica.cc:333] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c: stopping tablet replica
I20261120 03:02:54.114748   385 raft_consensus.cc:2440] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 3 LEADER]: Raft consensus shutting down.
I20261120 03:02:54.115278   385 raft_consensus.cc:2469] T 43123be7b9fc4ec2b0e23685d1605874 P a4e1c705ec064e7eb74a40c9a99fd34c [term 3 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:54.133369   385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:02:54.139866   385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:02:54.152577   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:02:54.153245   385 tablet_replica.cc:333] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b: stopping tablet replica
I20261120 03:02:54.153752   385 raft_consensus.cc:2440] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 3 FOLLOWER]: Raft consensus shutting down.
I20261120 03:02:54.154105   385 raft_consensus.cc:2469] T 43123be7b9fc4ec2b0e23685d1605874 P d2f8f09810044404b290ce30d0f28d4b [term 3 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:54.171200   385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:02:54.177789   385 master.cc:585] Master@127.0.96.126:43817 shutting down...
I20261120 03:02:54.190485   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:02:54.190889   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:02:54.191217   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 88e7003c263548a1a47d5399f68474db: stopping tablet replica
I20261120 03:02:54.207733   385 master.cc:607] Master@127.0.96.126:43817 shutdown complete.
[       OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (981 ms)
[----------] 3 tests from LeadershipChangeReportingTest (5346 ms total)

[----------] 4 tests from TsTabletManagerITest
[ RUN      ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20261120 03:02:54.229281   385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:44799
I20261120 03:02:54.230238   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:54.234601  1166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:54.235781  1167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:54.237306  1169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:54.237586   385 server_base.cc:1061] running on GCE node
I20261120 03:02:54.238300   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:54.238488   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:54.238648   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143774238633 us; error 0 us; skew 500 ppm
I20261120 03:02:54.239130   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:54.241142   385 webserver.cc:577] Webserver started at http://127.0.96.126:35679/ using document root <none> and password file <none>
I20261120 03:02:54.241587   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:54.241773   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:54.242013   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:54.242929   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "13b572e8c8814f5e946a5e885cb2d475"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.246845   385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20261120 03:02:54.249711  1174 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:54.250526   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20261120 03:02:54.250730   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "13b572e8c8814f5e946a5e885cb2d475"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.250938   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:54.267541   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:02:54.268543   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:54.300067   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:44799
I20261120 03:02:54.300164  1225 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:44799 every 8 connection(s)
I20261120 03:02:54.303516  1226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:54.312043  1226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475: Bootstrap starting.
I20261120 03:02:54.315793  1226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:54.319000  1226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475: No bootstrap required, opened a new log
I20261120 03:02:54.320672  1226 raft_consensus.cc:366] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b572e8c8814f5e946a5e885cb2d475" member_type: VOTER }
I20261120 03:02:54.321025  1226 raft_consensus.cc:392] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:02:54.321254  1226 raft_consensus.cc:787] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b572e8c8814f5e946a5e885cb2d475, State: Initialized, Role: FOLLOWER
I20261120 03:02:54.321837  1226 consensus_queue.cc:261] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b572e8c8814f5e946a5e885cb2d475" member_type: VOTER }
I20261120 03:02:54.322263  1226 raft_consensus.cc:407] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:02:54.322497  1226 raft_consensus.cc:534] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:02:54.322732  1226 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:54.323709  1226 raft_consensus.cc:556] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b572e8c8814f5e946a5e885cb2d475" member_type: VOTER }
I20261120 03:02:54.327623  1226 leader_election.cc:304] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13b572e8c8814f5e946a5e885cb2d475; no voters: 
I20261120 03:02:54.328213  1226 leader_election.cc:290] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:02:54.328286  1229 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:02:54.329769  1229 raft_consensus.cc:744] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 1 LEADER]: Becoming Leader. State: Replica: 13b572e8c8814f5e946a5e885cb2d475, State: Running, Role: LEADER
I20261120 03:02:54.330370  1229 consensus_queue.cc:238] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b572e8c8814f5e946a5e885cb2d475" member_type: VOTER }
I20261120 03:02:54.330857  1226 sys_catalog.cc:565] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:02:54.332979  1230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13b572e8c8814f5e946a5e885cb2d475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b572e8c8814f5e946a5e885cb2d475" member_type: VOTER } }
I20261120 03:02:54.333019  1231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13b572e8c8814f5e946a5e885cb2d475. Latest consensus state: current_term: 1 leader_uuid: "13b572e8c8814f5e946a5e885cb2d475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b572e8c8814f5e946a5e885cb2d475" member_type: VOTER } }
I20261120 03:02:54.333738  1231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:54.333770  1230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [sys.catalog]: This master's current role is: LEADER
I20261120 03:02:54.343549  1235 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:02:54.347476  1235 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:02:54.348378   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:02:54.354766  1235 catalog_manager.cc:1398] Generated new cluster ID: 4b9ccdd3131b4e28abee56f1dc1a8219
I20261120 03:02:54.354969  1235 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:02:54.376744  1235 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:02:54.377746  1235 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:02:54.389850  1235 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475: Generated new TSK 0
I20261120 03:02:54.390295  1235 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:02:54.413851   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:54.419052  1247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:54.419752  1248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:54.421631   385 server_base.cc:1061] running on GCE node
W20261120 03:02:54.421535  1250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:54.422421   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:54.422636   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:54.422806   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143774422790 us; error 0 us; skew 500 ppm
I20261120 03:02:54.423281   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:54.425338   385 webserver.cc:577] Webserver started at http://127.0.96.65:36049/ using document root <none> and password file <none>
I20261120 03:02:54.425792   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:54.425993   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:54.426239   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:54.427279   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "fb70d1328b754ce9afa3cc8507fe336c"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.431437   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.002s
I20261120 03:02:54.434484  1255 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:54.435268   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20261120 03:02:54.435482   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "fb70d1328b754ce9afa3cc8507fe336c"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.435712   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:54.457656   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:02:54.458464   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:54.459717   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:54.461700   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:54.461869   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:54.462049   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:54.462173   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:54.495426   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:36933
I20261120 03:02:54.495503  1317 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:36933 every 8 connection(s)
I20261120 03:02:54.499506   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:54.505822  1322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:54.506392  1323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:54.508775  1326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:54.508876   385 server_base.cc:1061] running on GCE node
I20261120 03:02:54.509826   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:54.510123   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:54.510337   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143774510323 us; error 0 us; skew 500 ppm
I20261120 03:02:54.510998   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:54.511476  1318 heartbeater.cc:345] Connected to a master server at 127.0.96.126:44799
I20261120 03:02:54.511842  1318 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:54.512600  1318 heartbeater.cc:508] Master 127.0.96.126:44799 requested a full tablet report, sending...
I20261120 03:02:54.513527   385 webserver.cc:577] Webserver started at http://127.0.96.66:38411/ using document root <none> and password file <none>
I20261120 03:02:54.514070   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:54.514312   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:54.514532  1191 ts_manager.cc:210] Registered new tserver with Master: fb70d1328b754ce9afa3cc8507fe336c (127.0.96.65:36933)
I20261120 03:02:54.514621   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:54.515997   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.516628  1191 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33774
I20261120 03:02:54.520097   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20261120 03:02:54.523138  1330 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:54.523777   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20261120 03:02:54.524014   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.524307   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:54.557129   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:02:54.557993   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:54.559185   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:54.561102   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:54.561272   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:54.561435   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:54.561555   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:54.595197   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:33381
I20261120 03:02:54.595307  1392 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:33381 every 8 connection(s)
I20261120 03:02:54.599495   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:02:54.605748  1396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:54.606714  1397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:02:54.609495  1399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:02:54.609799   385 server_base.cc:1061] running on GCE node
I20261120 03:02:54.610610   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:02:54.610821   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:02:54.611020   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143774611002 us; error 0 us; skew 500 ppm
I20261120 03:02:54.611620   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:02:54.612002  1393 heartbeater.cc:345] Connected to a master server at 127.0.96.126:44799
I20261120 03:02:54.612401  1393 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:54.613058  1393 heartbeater.cc:508] Master 127.0.96.126:44799 requested a full tablet report, sending...
I20261120 03:02:54.614274   385 webserver.cc:577] Webserver started at http://127.0.96.67:44599/ using document root <none> and password file <none>
I20261120 03:02:54.614737   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:02:54.614929   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:02:54.614892  1191 ts_manager.cc:210] Registered new tserver with Master: 83191dd6cf0a4e6ba32ec4d6b0baf8e1 (127.0.96.66:33381)
I20261120 03:02:54.615325   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:02:54.616328  1191 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33788
I20261120 03:02:54.616350   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-2-root/instance:
uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.620388   385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.001s
I20261120 03:02:54.623044  1404 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:54.623750   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I20261120 03:02:54.624014   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-2-root
uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c"
format_stamp: "Formatted at 2026-11-20 03:02:54 on dist-test-slave-54pf"
I20261120 03:02:54.624347   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1795143768723645-385-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:02:54.633543   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:02:54.634411   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:02:54.635551   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:02:54.637517   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:02:54.637713   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:54.637936   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:02:54.638094   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:02:54.694826   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.67:46697
I20261120 03:02:54.694900  1466 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.67:46697 every 8 connection(s)
I20261120 03:02:54.707515  1467 heartbeater.cc:345] Connected to a master server at 127.0.96.126:44799
I20261120 03:02:54.707840  1467 heartbeater.cc:462] Registering TS with master...
I20261120 03:02:54.708504  1467 heartbeater.cc:508] Master 127.0.96.126:44799 requested a full tablet report, sending...
I20261120 03:02:54.710311  1191 ts_manager.cc:210] Registered new tserver with Master: fd22d6a08b19497f8e0ba6f7730b2a1c (127.0.96.67:46697)
I20261120 03:02:54.710769   385 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011903127s
I20261120 03:02:54.711512  1191 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33794
I20261120 03:02:54.732864  1191 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33810:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
  }
}
W20261120 03:02:54.734082  1191 catalog_manager.cc:7244] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261120 03:02:54.779484  1432 tablet_service.cc:1495] Processing CreateTablet for tablet b16a610a167240dca15b363f5ce2376f (DEFAULT_TABLE table=test-table [id=1f6fb7e133e443d480e0f8d0e95bf9f0]), partition=
I20261120 03:02:54.781028  1432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16a610a167240dca15b363f5ce2376f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:54.780997  1283 tablet_service.cc:1495] Processing CreateTablet for tablet b16a610a167240dca15b363f5ce2376f (DEFAULT_TABLE table=test-table [id=1f6fb7e133e443d480e0f8d0e95bf9f0]), partition=
I20261120 03:02:54.782060  1283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16a610a167240dca15b363f5ce2376f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:54.783007  1358 tablet_service.cc:1495] Processing CreateTablet for tablet b16a610a167240dca15b363f5ce2376f (DEFAULT_TABLE table=test-table [id=1f6fb7e133e443d480e0f8d0e95bf9f0]), partition=
I20261120 03:02:54.784435  1358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16a610a167240dca15b363f5ce2376f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:02:54.797142  1482 tablet_bootstrap.cc:492] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c: Bootstrap starting.
I20261120 03:02:54.799448  1483 tablet_bootstrap.cc:492] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c: Bootstrap starting.
I20261120 03:02:54.803227  1484 tablet_bootstrap.cc:492] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1: Bootstrap starting.
I20261120 03:02:54.806152  1482 tablet_bootstrap.cc:654] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:54.807794  1483 tablet_bootstrap.cc:654] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:54.808522  1484 tablet_bootstrap.cc:654] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1: Neither blocks nor log segments found. Creating new log.
I20261120 03:02:54.813802  1482 tablet_bootstrap.cc:492] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c: No bootstrap required, opened a new log
I20261120 03:02:54.814944  1482 ts_tablet_manager.cc:1449] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c: Time spent bootstrapping tablet: real 0.018s	user 0.011s	sys 0.005s
I20261120 03:02:54.815907  1483 tablet_bootstrap.cc:492] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c: No bootstrap required, opened a new log
I20261120 03:02:54.816356  1483 ts_tablet_manager.cc:1449] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c: Time spent bootstrapping tablet: real 0.017s	user 0.006s	sys 0.009s
I20261120 03:02:54.818130  1484 tablet_bootstrap.cc:492] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1: No bootstrap required, opened a new log
I20261120 03:02:54.817884  1482 raft_consensus.cc:366] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:54.818554  1484 ts_tablet_manager.cc:1449] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1: Time spent bootstrapping tablet: real 0.016s	user 0.008s	sys 0.005s
I20261120 03:02:54.818588  1482 raft_consensus.cc:392] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:02:54.818959  1482 raft_consensus.cc:787] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd22d6a08b19497f8e0ba6f7730b2a1c, State: Initialized, Role: FOLLOWER
I20261120 03:02:54.819274  1483 raft_consensus.cc:366] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:54.819559  1482 consensus_queue.cc:261] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:54.820026  1483 raft_consensus.cc:392] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:02:54.820377  1483 raft_consensus.cc:787] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb70d1328b754ce9afa3cc8507fe336c, State: Initialized, Role: FOLLOWER
I20261120 03:02:54.821105  1483 consensus_queue.cc:261] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:54.822284  1467 heartbeater.cc:500] Master 127.0.96.126:44799 was elected leader, sending a full tablet report...
I20261120 03:02:54.821482  1484 raft_consensus.cc:366] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:54.823068  1482 ts_tablet_manager.cc:1480] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20261120 03:02:54.823046  1484 raft_consensus.cc:392] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:02:54.823529  1484 raft_consensus.cc:787] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83191dd6cf0a4e6ba32ec4d6b0baf8e1, State: Initialized, Role: FOLLOWER
I20261120 03:02:54.825052  1318 heartbeater.cc:500] Master 127.0.96.126:44799 was elected leader, sending a full tablet report...
I20261120 03:02:54.825153  1484 consensus_queue.cc:261] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:54.827224  1483 ts_tablet_manager.cc:1480] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.002s
I20261120 03:02:54.829017  1393 heartbeater.cc:500] Master 127.0.96.126:44799 was elected leader, sending a full tablet report...
I20261120 03:02:54.829488  1484 ts_tablet_manager.cc:1480] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1: Time spent starting tablet: real 0.011s	user 0.001s	sys 0.008s
I20261120 03:02:55.010123  1488 raft_consensus.cc:534] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:02:55.010795  1488 raft_consensus.cc:556] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:55.012936  1488 leader_election.cc:290] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb70d1328b754ce9afa3cc8507fe336c (127.0.96.65:36933), 83191dd6cf0a4e6ba32ec4d6b0baf8e1 (127.0.96.66:33381)
I20261120 03:02:55.021757  1293 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "b16a610a167240dca15b363f5ce2376f" candidate_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb70d1328b754ce9afa3cc8507fe336c" is_pre_election: true
I20261120 03:02:55.022501  1293 raft_consensus.cc:2062] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate fd22d6a08b19497f8e0ba6f7730b2a1c in term 0
I20261120 03:02:55.023605  1368 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "b16a610a167240dca15b363f5ce2376f" candidate_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" is_pre_election: true
I20261120 03:02:55.023746  1405 leader_election.cc:304] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fb70d1328b754ce9afa3cc8507fe336c, fd22d6a08b19497f8e0ba6f7730b2a1c; no voters: 
I20261120 03:02:55.024391  1368 raft_consensus.cc:2062] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate fd22d6a08b19497f8e0ba6f7730b2a1c in term 0
I20261120 03:02:55.024446  1488 raft_consensus.cc:3024] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:02:55.024843  1488 raft_consensus.cc:534] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:02:55.025144  1488 raft_consensus.cc:3296] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:55.026297  1488 raft_consensus.cc:556] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:55.031213  1488 leader_election.cc:290] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [CANDIDATE]: Term 1 election: Requested vote from peers fb70d1328b754ce9afa3cc8507fe336c (127.0.96.65:36933), 83191dd6cf0a4e6ba32ec4d6b0baf8e1 (127.0.96.66:33381)
I20261120 03:02:55.032352  1293 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "b16a610a167240dca15b363f5ce2376f" candidate_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb70d1328b754ce9afa3cc8507fe336c"
I20261120 03:02:55.032431  1368 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "b16a610a167240dca15b363f5ce2376f" candidate_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1"
I20261120 03:02:55.032927  1293 raft_consensus.cc:3296] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:55.033008  1368 raft_consensus.cc:3296] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:02:55.033754  1293 raft_consensus.cc:2062] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate fd22d6a08b19497f8e0ba6f7730b2a1c in term 1
I20261120 03:02:55.033829  1368 raft_consensus.cc:2062] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate fd22d6a08b19497f8e0ba6f7730b2a1c in term 1
I20261120 03:02:55.038827  1405 leader_election.cc:304] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fb70d1328b754ce9afa3cc8507fe336c, fd22d6a08b19497f8e0ba6f7730b2a1c; no voters: 
I20261120 03:02:55.039443  1488 raft_consensus.cc:3024] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:02:55.039865  1488 raft_consensus.cc:744] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 1 LEADER]: Becoming Leader. State: Replica: fd22d6a08b19497f8e0ba6f7730b2a1c, State: Running, Role: LEADER
I20261120 03:02:55.040616  1488 consensus_queue.cc:238] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } }
I20261120 03:02:55.047194  1488 consensus_queue.cc:1049] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:02:55.048202  1492 consensus_queue.cc:1049] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:02:55.048311  1191 catalog_manager.cc:5856] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c reported cstate change: term changed from 0 to 1, leader changed from <none> to fd22d6a08b19497f8e0ba6f7730b2a1c (127.0.96.67). New cstate: current_term: 1 leader_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb70d1328b754ce9afa3cc8507fe336c" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 36933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd22d6a08b19497f8e0ba6f7730b2a1c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83191dd6cf0a4e6ba32ec4d6b0baf8e1" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 33381 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:00.121285   385 tablet_replica.cc:333] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c: stopping tablet replica
I20261120 03:03:00.121924   385 raft_consensus.cc:2440] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 1 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:00.122460   385 raft_consensus.cc:2469] T b16a610a167240dca15b363f5ce2376f P fb70d1328b754ce9afa3cc8507fe336c [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:00.373221  1405 consensus_queue.cc:580] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [LEADER]: The tablet replica hosted on peer fb70d1328b754ce9afa3cc8507fe336c has failed
W20261120 03:03:00.373754  1405 consensus_peers.cc:597] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c -> Peer fb70d1328b754ce9afa3cc8507fe336c (127.0.96.65:36933): Couldn't send request to peer fb70d1328b754ce9afa3cc8507fe336c. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:00.390484   385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:00.412593   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:00.429608   385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:00.438791   385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:03:00.455166   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:00.455706   385 tablet_replica.cc:333] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1: stopping tablet replica
I20261120 03:03:00.456195   385 raft_consensus.cc:2440] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:00.456574   385 raft_consensus.cc:2469] T b16a610a167240dca15b363f5ce2376f P 83191dd6cf0a4e6ba32ec4d6b0baf8e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:00.473994   385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:03:00.482470   385 tablet_server.cc:179] TabletServer@127.0.96.67:0 shutting down...
I20261120 03:03:00.499900   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:00.500712   385 tablet_replica.cc:333] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c: stopping tablet replica
I20261120 03:03:00.501191   385 raft_consensus.cc:2440] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:00.501853   385 raft_consensus.cc:2469] T b16a610a167240dca15b363f5ce2376f P fd22d6a08b19497f8e0ba6f7730b2a1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:00.518859   385 tablet_server.cc:196] TabletServer@127.0.96.67:0 shutdown complete.
I20261120 03:03:00.526975   385 master.cc:585] Master@127.0.96.126:44799 shutting down...
I20261120 03:03:00.544088   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:00.544662   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:00.544971   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 13b572e8c8814f5e946a5e885cb2d475: stopping tablet replica
I20261120 03:03:00.561723   385 master.cc:607] Master@127.0.96.126:44799 shutdown complete.
[       OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (6362 ms)
[ RUN      ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20261120 03:03:00.591781   385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:38541
I20261120 03:03:00.592882   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:00.597900  1509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:00.598313  1510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:00.599647  1512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:00.600333   385 server_base.cc:1061] running on GCE node
I20261120 03:03:00.601065   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:00.601217   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:00.601384   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143780601359 us; error 0 us; skew 500 ppm
I20261120 03:03:00.601859   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:00.603868   385 webserver.cc:577] Webserver started at http://127.0.96.126:37539/ using document root <none> and password file <none>
I20261120 03:03:00.604403   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:00.604562   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:00.604809   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:00.605844   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "c9caea48f0ac4663b52f5b498561e813"
format_stamp: "Formatted at 2026-11-20 03:03:00 on dist-test-slave-54pf"
I20261120 03:03:00.609854   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20261120 03:03:00.612886  1517 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:00.613592   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20261120 03:03:00.613847   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "c9caea48f0ac4663b52f5b498561e813"
format_stamp: "Formatted at 2026-11-20 03:03:00 on dist-test-slave-54pf"
I20261120 03:03:00.614109   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:00.635443   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:00.636543   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:00.670980   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:38541
I20261120 03:03:00.671065  1568 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:38541 every 8 connection(s)
I20261120 03:03:00.674728  1569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:00.684383  1569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813: Bootstrap starting.
I20261120 03:03:00.688879  1569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:00.692813  1569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813: No bootstrap required, opened a new log
I20261120 03:03:00.694752  1569 raft_consensus.cc:366] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9caea48f0ac4663b52f5b498561e813" member_type: VOTER }
I20261120 03:03:00.695129  1569 raft_consensus.cc:392] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:00.695348  1569 raft_consensus.cc:787] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9caea48f0ac4663b52f5b498561e813, State: Initialized, Role: FOLLOWER
I20261120 03:03:00.695856  1569 consensus_queue.cc:261] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9caea48f0ac4663b52f5b498561e813" member_type: VOTER }
I20261120 03:03:00.696308  1569 raft_consensus.cc:407] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:00.696532  1569 raft_consensus.cc:534] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:00.696786  1569 raft_consensus.cc:3296] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:00.697757  1569 raft_consensus.cc:556] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9caea48f0ac4663b52f5b498561e813" member_type: VOTER }
I20261120 03:03:00.701982  1569 leader_election.cc:304] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9caea48f0ac4663b52f5b498561e813; no voters: 
I20261120 03:03:00.702471  1569 leader_election.cc:290] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:03:00.702559  1572 raft_consensus.cc:3024] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:00.703918  1572 raft_consensus.cc:744] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 1 LEADER]: Becoming Leader. State: Replica: c9caea48f0ac4663b52f5b498561e813, State: Running, Role: LEADER
I20261120 03:03:00.704583  1572 consensus_queue.cc:238] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9caea48f0ac4663b52f5b498561e813" member_type: VOTER }
I20261120 03:03:00.705353  1569 sys_catalog.cc:565] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:00.706705  1573 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9caea48f0ac4663b52f5b498561e813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9caea48f0ac4663b52f5b498561e813" member_type: VOTER } }
I20261120 03:03:00.707118  1573 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:00.707831  1574 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9caea48f0ac4663b52f5b498561e813. Latest consensus state: current_term: 1 leader_uuid: "c9caea48f0ac4663b52f5b498561e813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9caea48f0ac4663b52f5b498561e813" member_type: VOTER } }
I20261120 03:03:00.708557  1574 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:00.710824  1576 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:00.715314  1576 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:00.720172   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:00.723467  1576 catalog_manager.cc:1398] Generated new cluster ID: 8c829e260350433bb29d36fa6ca10d3d
I20261120 03:03:00.723735  1576 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:00.737931  1576 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:00.739041  1576 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:00.752295  1576 catalog_manager.cc:6237] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813: Generated new TSK 0
I20261120 03:03:00.752805  1576 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:00.786230   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:00.791600  1590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:00.792630  1591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:00.794157  1593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:00.794343   385 server_base.cc:1061] running on GCE node
I20261120 03:03:00.795176   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:00.795384   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:00.795560   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143780795541 us; error 0 us; skew 500 ppm
I20261120 03:03:00.796051   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:00.797953   385 webserver.cc:577] Webserver started at http://127.0.96.65:34477/ using document root <none> and password file <none>
I20261120 03:03:00.798414   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:00.798621   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:00.798851   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:00.799840   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "510ca5f2907b4b9e8d66c9e553be512c"
format_stamp: "Formatted at 2026-11-20 03:03:00 on dist-test-slave-54pf"
I20261120 03:03:00.803959   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20261120 03:03:00.806663  1598 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:00.807471   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20261120 03:03:00.807756   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "510ca5f2907b4b9e8d66c9e553be512c"
format_stamp: "Formatted at 2026-11-20 03:03:00 on dist-test-slave-54pf"
I20261120 03:03:00.808044   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:00.823899   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:00.824952   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:00.826256   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:00.828415   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:00.828616   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:00.828858   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:00.829046   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:00.866124   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:37041
I20261120 03:03:00.866219  1660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:37041 every 8 connection(s)
I20261120 03:03:00.887826  1661 heartbeater.cc:345] Connected to a master server at 127.0.96.126:38541
I20261120 03:03:00.888103  1661 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:00.888728  1661 heartbeater.cc:508] Master 127.0.96.126:38541 requested a full tablet report, sending...
I20261120 03:03:00.890677  1534 ts_manager.cc:210] Registered new tserver with Master: 510ca5f2907b4b9e8d66c9e553be512c (127.0.96.65:37041)
I20261120 03:03:00.890942   385 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.021797183s
I20261120 03:03:00.892396  1534 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35694
I20261120 03:03:00.908631   385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:00.921871   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:00.936878   385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:00.943432   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261120 03:03:00.945909   385 tablet_server.cc:94] deduplicated master addresses: 127.0.96.126:38541
W20261120 03:03:00.948938  1671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:00.949092  1672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:00.950181  1674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:00.951167   385 server_base.cc:1061] running on GCE node
I20261120 03:03:00.951890   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:00.952099   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:00.952296   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143780952284 us; error 0 us; skew 500 ppm
I20261120 03:03:00.952708   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:00.954696   385 webserver.cc:577] Webserver started at http://127.0.96.65:34477/ using document root <none> and password file <none>
I20261120 03:03:00.955101   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:00.955242   385 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261120 03:03:00.958850   385 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20261120 03:03:00.961122  1679 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:00.961823   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I20261120 03:03:00.962112   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "510ca5f2907b4b9e8d66c9e553be512c"
format_stamp: "Formatted at 2026-11-20 03:03:00 on dist-test-slave-54pf"
I20261120 03:03:00.962409   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:00.976797   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:00.977682   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:00.978928   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:00.980959   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:00.981195   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:00.981401   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:00.981571   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:01.015816   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:37041
I20261120 03:03:01.015898  1741 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:37041 every 8 connection(s)
I20261120 03:03:01.023146   385 tablet_server.cc:179] TabletServer@127.0.96.65:37041 shutting down...
I20261120 03:03:01.032740  1742 heartbeater.cc:345] Connected to a master server at 127.0.96.126:38541
I20261120 03:03:01.033221  1742 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:01.034045  1742 heartbeater.cc:508] Master 127.0.96.126:38541 requested a full tablet report, sending...
I20261120 03:03:01.036316  1534 ts_manager.cc:210] Re-registered known tserver with Master: 510ca5f2907b4b9e8d66c9e553be512c (127.0.96.65:37041)
I20261120 03:03:01.040264  1534 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35712
I20261120 03:03:01.042716   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:01.057706   385 tablet_server.cc:196] TabletServer@127.0.96.65:37041 shutdown complete.
I20261120 03:03:01.063872   385 master.cc:585] Master@127.0.96.126:38541 shutting down...
I20261120 03:03:01.075528   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:01.075894   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:01.076334   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P c9caea48f0ac4663b52f5b498561e813: stopping tablet replica
I20261120 03:03:01.093824   385 master.cc:607] Master@127.0.96.126:38541 shutdown complete.
[       OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (521 ms)
[ RUN      ] TsTabletManagerITest.TestTableStats
I20261120 03:03:01.113286   385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:43725,127.0.96.125:41159,127.0.96.124:46415
I20261120 03:03:01.114382   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:01.118534  1749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:01.119565  1750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.120769   385 server_base.cc:1061] running on GCE node
W20261120 03:03:01.121533  1752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.122123   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:01.122313   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:01.122448   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143781122433 us; error 0 us; skew 500 ppm
I20261120 03:03:01.122884   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:01.124884   385 webserver.cc:577] Webserver started at http://127.0.96.126:38517/ using document root <none> and password file <none>
I20261120 03:03:01.125314   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:01.125489   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:01.125732   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:01.126621   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "764428dd82184375a0c80cbf46397e89"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.130623   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.003s
I20261120 03:03:01.133477  1757 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:01.134092   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20261120 03:03:01.134356   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "764428dd82184375a0c80cbf46397e89"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.134619   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:01.172819   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:01.173703   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:01.204849   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:43725
I20261120 03:03:01.204936  1808 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:43725 every 8 connection(s)
I20261120 03:03:01.208185  1809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:01.208460   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:01.213310  1811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:01.214320  1812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.216320  1809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
W20261120 03:03:01.216814  1814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.217439   385 server_base.cc:1061] running on GCE node
I20261120 03:03:01.218364   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:01.218664   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:01.218957   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143781218931 us; error 0 us; skew 500 ppm
I20261120 03:03:01.219802   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:01.222317   385 webserver.cc:577] Webserver started at http://127.0.96.125:43061/ using document root <none> and password file <none>
I20261120 03:03:01.222898   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:01.223140   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:01.223419   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:01.224553   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root/instance:
uuid: "e68c557e0a6047f5956737cfd33fffb5"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.228746  1809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
I20261120 03:03:01.229902   385 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
W20261120 03:03:01.231052  1761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.96.125:41159: connect: Connection refused (error 111)
I20261120 03:03:01.233788  1822 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
W20261120 03:03:01.234326  1809 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.96.125:41159: Network error: Client connection negotiation failed: client connection to 127.0.96.125:41159: connect: Connection refused (error 111)
I20261120 03:03:01.234531   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20261120 03:03:01.234881   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root
uuid: "e68c557e0a6047f5956737cfd33fffb5"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.235140   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:01.253479   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:01.254280   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:01.285888  1809 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } attempt: 1
I20261120 03:03:01.286357   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.125:41159
I20261120 03:03:01.286491  1873 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.125:41159 every 8 connection(s)
I20261120 03:03:01.290812   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261120 03:03:01.290877  1875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20261120 03:03:01.297047  1877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:01.298614  1878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.300428  1809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } has no permanent_uuid. Determining permanent_uuid...
I20261120 03:03:01.301977  1875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
I20261120 03:03:01.302690   385 server_base.cc:1061] running on GCE node
W20261120 03:03:01.303306  1880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.304421   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:01.304821   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:01.305078   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143781305058 us; error 0 us; skew 500 ppm
I20261120 03:03:01.305842   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20261120 03:03:01.307145  1809 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.96.124:46415: Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111)
I20261120 03:03:01.308540   385 webserver.cc:577] Webserver started at http://127.0.96.124:36403/ using document root <none> and password file <none>
I20261120 03:03:01.309111   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:01.309319   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:01.309609   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:01.310768   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root/instance:
uuid: "3a4598f24f6246008d66f45cff9355af"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.314247  1875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
I20261120 03:03:01.316304   385 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.003s
I20261120 03:03:01.320317  1887 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:01.321255   385 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20261120 03:03:01.321619   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root
uuid: "3a4598f24f6246008d66f45cff9355af"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.321916   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:01.324059  1875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } has no permanent_uuid. Determining permanent_uuid...
W20261120 03:03:01.329483  1875 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.96.124:46415: Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111)
I20261120 03:03:01.335871   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:01.336994   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:01.347918  1875 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } attempt: 1
I20261120 03:03:01.352564  1809 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } attempt: 1
W20261120 03:03:01.353295  1875 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.96.124:46415: Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111)
W20261120 03:03:01.357419  1809 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.96.124:46415: Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111)
I20261120 03:03:01.374946   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.124:46415
I20261120 03:03:01.375042  1939 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.124:46415 every 8 connection(s)
I20261120 03:03:01.377769   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:01.378423  1940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:01.384023  1940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
I20261120 03:03:01.394284  1940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
I20261120 03:03:01.397851  1809 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } attempt: 2
I20261120 03:03:01.403720  1940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } has no permanent_uuid. Determining permanent_uuid...
I20261120 03:03:01.407793  1875 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } attempt: 2
I20261120 03:03:01.417574  1809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: Bootstrap starting.
I20261120 03:03:01.421790  1940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: Bootstrap starting.
I20261120 03:03:01.422850  1809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:01.426923  1940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:01.427493  1875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: Bootstrap starting.
I20261120 03:03:01.427958  1809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: No bootstrap required, opened a new log
I20261120 03:03:01.431100  1940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: No bootstrap required, opened a new log
I20261120 03:03:01.430864  1809 raft_consensus.cc:366] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.431659  1809 raft_consensus.cc:392] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:01.431965  1809 raft_consensus.cc:787] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 764428dd82184375a0c80cbf46397e89, State: Initialized, Role: FOLLOWER
I20261120 03:03:01.432710  1809 consensus_queue.cc:261] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.433588  1940 raft_consensus.cc:366] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.434165  1940 raft_consensus.cc:392] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:01.434300  1875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:01.434554  1940 raft_consensus.cc:787] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a4598f24f6246008d66f45cff9355af, State: Initialized, Role: FOLLOWER
I20261120 03:03:01.434718  1947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.435432  1947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:01.435432  1940 consensus_queue.cc:261] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.436347  1809 sys_catalog.cc:565] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:01.437691  1948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.438521  1948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:01.439214  1940 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:01.446386  1875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: No bootstrap required, opened a new log
I20261120 03:03:01.449472  1875 raft_consensus.cc:366] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.450191  1875 raft_consensus.cc:392] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:01.450466  1875 raft_consensus.cc:787] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68c557e0a6047f5956737cfd33fffb5, State: Initialized, Role: FOLLOWER
I20261120 03:03:01.451133  1875 consensus_queue.cc:261] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.457391   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20261120 03:03:01.459287  1965 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.459831  1965 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:01.462009  1875 sys_catalog.cc:565] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: configured and running, proceeding with master startup.
W20261120 03:03:01.462469  1969 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20261120 03:03:01.462754  1969 catalog_manager.cc:898] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20261120 03:03:01.466526  1972 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20261120 03:03:01.466873  1972 catalog_manager.cc:898] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20261120 03:03:01.472626   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20261120 03:03:01.473889   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:01.475297  1983 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
E20261120 03:03:01.475345   385 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20261120 03:03:01.475687  1983 catalog_manager.cc:898] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20261120 03:03:01.480404  1984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:01.480737  1985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:01.482128  1987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.482893   385 server_base.cc:1061] running on GCE node
I20261120 03:03:01.483560   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:01.483754   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:01.483917   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143781483902 us; error 0 us; skew 500 ppm
I20261120 03:03:01.484443   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:01.486449   385 webserver.cc:577] Webserver started at http://127.0.96.65:44329/ using document root <none> and password file <none>
I20261120 03:03:01.486902   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:01.487088   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:01.487344   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:01.488432   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.492588   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20261120 03:03:01.495311  1992 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:01.495929   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20261120 03:03:01.496245   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.496510   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:01.515136  1947 raft_consensus.cc:534] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:01.515570  1947 raft_consensus.cc:556] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.517510   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:01.517567  1947 leader_election.cc:290] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:01.518203  1849 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "764428dd82184375a0c80cbf46397e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e68c557e0a6047f5956737cfd33fffb5" is_pre_election: true
I20261120 03:03:01.518958  1849 raft_consensus.cc:2062] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 764428dd82184375a0c80cbf46397e89 in term 0
I20261120 03:03:01.518705  1915 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "764428dd82184375a0c80cbf46397e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af" is_pre_election: true
I20261120 03:03:01.519145   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:01.519536  1915 raft_consensus.cc:2062] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 764428dd82184375a0c80cbf46397e89 in term 0
I20261120 03:03:01.519924  1761 leader_election.cc:304] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 764428dd82184375a0c80cbf46397e89, e68c557e0a6047f5956737cfd33fffb5; no voters: 
I20261120 03:03:01.520692  1947 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:03:01.521049  1947 raft_consensus.cc:534] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:01.521186   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:01.521420  1947 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:01.522930  1947 raft_consensus.cc:556] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.523677   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:01.523969   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:01.524252   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:01.524459   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:01.528769  1947 leader_election.cc:290] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 1 election: Requested vote from peers e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:01.529448  1849 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "764428dd82184375a0c80cbf46397e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e68c557e0a6047f5956737cfd33fffb5"
I20261120 03:03:01.529816  1915 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "764428dd82184375a0c80cbf46397e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af"
I20261120 03:03:01.529968  1849 raft_consensus.cc:3296] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:01.530480  1915 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:01.531128  1849 raft_consensus.cc:2062] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 764428dd82184375a0c80cbf46397e89 in term 1
I20261120 03:03:01.531443  1915 raft_consensus.cc:2062] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 764428dd82184375a0c80cbf46397e89 in term 1
I20261120 03:03:01.538159  1760 leader_election.cc:304] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89; no voters: 
I20261120 03:03:01.539101  1947 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:01.539546  1947 raft_consensus.cc:744] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 1 LEADER]: Becoming Leader. State: Replica: 764428dd82184375a0c80cbf46397e89, State: Running, Role: LEADER
I20261120 03:03:01.540349  1947 consensus_queue.cc:238] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:01.545004  1998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764428dd82184375a0c80cbf46397e89. Latest consensus state: current_term: 1 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.545918  1998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:01.549432  2015 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:01.551254  2014 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261120 03:03:01.552515  1998 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:01.557214  1948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764428dd82184375a0c80cbf46397e89. Latest consensus state: current_term: 1 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.557950  1948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:01.560458  1965 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764428dd82184375a0c80cbf46397e89. Latest consensus state: current_term: 1 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.561151  1965 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:01.561973  2015 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:01.568854  1948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.569548  1948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:01.573836  1947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.575049  1947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:01.576162  2014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.577209  2014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:01.579603  1965 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:01.580376  1965 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:01.609448  2015 catalog_manager.cc:1398] Generated new cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:01.609782  2015 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:01.640226   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:40485
I20261120 03:03:01.640429  2067 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:40485 every 8 connection(s)
I20261120 03:03:01.661283  2015 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:01.663110  2015 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:01.663178  2068 heartbeater.cc:345] Connected to a master server at 127.0.96.124:46415
I20261120 03:03:01.667663  2068 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:01.668567  2068 heartbeater.cc:508] Master 127.0.96.124:46415 requested a full tablet report, sending...
I20261120 03:03:01.671627  1905 ts_manager.cc:210] Registered new tserver with Master: 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:01.674073  2069 heartbeater.cc:345] Connected to a master server at 127.0.96.125:41159
I20261120 03:03:01.674381  2069 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:01.675002  2069 heartbeater.cc:508] Master 127.0.96.125:41159 requested a full tablet report, sending...
I20261120 03:03:01.677265  1839 ts_manager.cc:210] Registered new tserver with Master: 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:01.681150   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261120 03:03:01.683161   385 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20261120 03:03:01.685400  2070 heartbeater.cc:345] Connected to a master server at 127.0.96.126:43725
I20261120 03:03:01.685720  2070 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:01.686367  2070 heartbeater.cc:508] Master 127.0.96.126:43725 requested a full tablet report, sending...
I20261120 03:03:01.689021  1774 ts_manager.cc:210] Registered new tserver with Master: 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
W20261120 03:03:01.690783  2075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:01.697875  2076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.698508   385 server_base.cc:1061] running on GCE node
W20261120 03:03:01.699224  2078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.700006  2015 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: Generated new TSK 0
I20261120 03:03:01.700690  2015 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:01.700641   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:01.701041   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:01.701247   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143781701231 us; error 0 us; skew 500 ppm
I20261120 03:03:01.701924   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:01.709890   385 webserver.cc:577] Webserver started at http://127.0.96.66:35965/ using document root <none> and password file <none>
I20261120 03:03:01.710428   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:01.710634   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:01.710883   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:01.711920   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "a157015bf640485fb805159465a9ab1b"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.716527   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.002s
I20261120 03:03:01.719484  2083 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:01.720330   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20261120 03:03:01.720567   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "a157015bf640485fb805159465a9ab1b"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.720791   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:01.738719   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:01.739830   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:01.741143   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:01.743608   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:01.743829   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:01.744057   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:01.744309   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20261120 03:03:01.800279  1774 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41076
I20261120 03:03:01.822762   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:36367
I20261120 03:03:01.822841  2145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:36367 every 8 connection(s)
I20261120 03:03:01.847575   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261120 03:03:01.850085   385 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20261120 03:03:01.859768  2153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.860879  2146 heartbeater.cc:345] Connected to a master server at 127.0.96.124:46415
I20261120 03:03:01.861263  2146 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:01.862094  2146 heartbeater.cc:508] Master 127.0.96.124:46415 requested a full tablet report, sending...
W20261120 03:03:01.864267  2154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.864868  1905 ts_manager.cc:210] Registered new tserver with Master: a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:01.869925  2147 heartbeater.cc:345] Connected to a master server at 127.0.96.125:41159
I20261120 03:03:01.870450  2147 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:01.872058  2147 heartbeater.cc:508] Master 127.0.96.125:41159 requested a full tablet report, sending...
I20261120 03:03:01.873178  2148 heartbeater.cc:345] Connected to a master server at 127.0.96.126:43725
I20261120 03:03:01.873575  2148 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:01.873746   385 server_base.cc:1061] running on GCE node
I20261120 03:03:01.874464  2148 heartbeater.cc:508] Master 127.0.96.126:43725 requested a full tablet report, sending...
I20261120 03:03:01.875304  1839 ts_manager.cc:210] Registered new tserver with Master: a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
W20261120 03:03:01.875456  2156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:01.876552   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20261120 03:03:01.876696  1774 ts_manager.cc:210] Registered new tserver with Master: a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
W20261120 03:03:01.876863   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:01.877101   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143781877083 us; error 0 us; skew 500 ppm
I20261120 03:03:01.877864   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:01.878427  1774 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41092
I20261120 03:03:01.881500   385 webserver.cc:577] Webserver started at http://127.0.96.67:42435/ using document root <none> and password file <none>
I20261120 03:03:01.881984   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:01.882174   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:01.882433   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:01.883646   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-2-root/instance:
uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.888072   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20261120 03:03:01.891036  2161 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:01.891723   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20261120 03:03:01.892037   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-2-root
uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:01.892364   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:01.902702  2070 heartbeater.cc:500] Master 127.0.96.126:43725 was elected leader, sending a full tablet report...
I20261120 03:03:01.904440   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:01.905884   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:01.907900   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:01.909979   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:01.910190   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:01.910494   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:01.910652   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:01.981515  2148 heartbeater.cc:500] Master 127.0.96.126:43725 was elected leader, sending a full tablet report...
I20261120 03:03:01.991431   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.67:35169
I20261120 03:03:01.991528  2223 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.67:35169 every 8 connection(s)
I20261120 03:03:02.014479  2224 heartbeater.cc:345] Connected to a master server at 127.0.96.124:46415
I20261120 03:03:02.014946  2224 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:02.015379  2225 heartbeater.cc:345] Connected to a master server at 127.0.96.125:41159
I20261120 03:03:02.015771  2225 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:02.015767  2224 heartbeater.cc:508] Master 127.0.96.124:46415 requested a full tablet report, sending...
I20261120 03:03:02.016768  2225 heartbeater.cc:508] Master 127.0.96.125:41159 requested a full tablet report, sending...
I20261120 03:03:02.019119  1905 ts_manager.cc:210] Registered new tserver with Master: 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.019138  1839 ts_manager.cc:210] Registered new tserver with Master: 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.020800  2226 heartbeater.cc:345] Connected to a master server at 127.0.96.126:43725
I20261120 03:03:02.021080  2226 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:02.021682  2226 heartbeater.cc:508] Master 127.0.96.126:43725 requested a full tablet report, sending...
I20261120 03:03:02.023357  1774 ts_manager.cc:210] Registered new tserver with Master: 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.023622   385 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019925065s
I20261120 03:03:02.025103  1774 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41106
I20261120 03:03:02.057925  1774 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41110:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20261120 03:03:02.060530  1774 catalog_manager.cc:7244] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261120 03:03:02.127462  2226 heartbeater.cc:500] Master 127.0.96.126:43725 was elected leader, sending a full tablet report...
I20261120 03:03:02.151404  2030 tablet_service.cc:1495] Processing CreateTablet for tablet 791003cb42cf4e50a7666946f7ffad6b (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.152942  2030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 791003cb42cf4e50a7666946f7ffad6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.153169  2189 tablet_service.cc:1495] Processing CreateTablet for tablet 791003cb42cf4e50a7666946f7ffad6b (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.154213  2188 tablet_service.cc:1495] Processing CreateTablet for tablet ff883ba4b464463a9628eff440f1190a (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.154630  2189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 791003cb42cf4e50a7666946f7ffad6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.155472  2188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff883ba4b464463a9628eff440f1190a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.155555  2029 tablet_service.cc:1495] Processing CreateTablet for tablet ff883ba4b464463a9628eff440f1190a (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.157862  2029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff883ba4b464463a9628eff440f1190a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.151424  2028 tablet_service.cc:1495] Processing CreateTablet for tablet 178ee4361c534583b29d93a1e67d3c4f (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.158679  2187 tablet_service.cc:1495] Processing CreateTablet for tablet 178ee4361c534583b29d93a1e67d3c4f (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.162472  2187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178ee4361c534583b29d93a1e67d3c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.163736  2028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178ee4361c534583b29d93a1e67d3c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.163647  2111 tablet_service.cc:1495] Processing CreateTablet for tablet 791003cb42cf4e50a7666946f7ffad6b (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.165644  2110 tablet_service.cc:1495] Processing CreateTablet for tablet ff883ba4b464463a9628eff440f1190a (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.167827  2111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 791003cb42cf4e50a7666946f7ffad6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.167251  2109 tablet_service.cc:1495] Processing CreateTablet for tablet 178ee4361c534583b29d93a1e67d3c4f (DEFAULT_TABLE table=test-table [id=267fd8b91bac4f058896b2ba899401ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:02.170498  2110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff883ba4b464463a9628eff440f1190a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.171813  2109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178ee4361c534583b29d93a1e67d3c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:02.205672  2246 tablet_bootstrap.cc:492] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: Bootstrap starting.
I20261120 03:03:02.208734  2245 tablet_bootstrap.cc:492] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: Bootstrap starting.
I20261120 03:03:02.229111  2245 tablet_bootstrap.cc:654] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.231546  2247 tablet_bootstrap.cc:492] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Bootstrap starting.
I20261120 03:03:02.233209  2246 tablet_bootstrap.cc:654] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.238560  2245 tablet_bootstrap.cc:492] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: No bootstrap required, opened a new log
I20261120 03:03:02.239066  2245 ts_tablet_manager.cc:1449] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: Time spent bootstrapping tablet: real 0.033s	user 0.006s	sys 0.013s
I20261120 03:03:02.242333  2245 raft_consensus.cc:366] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.244211  2245 raft_consensus.cc:392] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.244576  2245 raft_consensus.cc:787] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.245257  2245 consensus_queue.cc:261] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.246409  2247 tablet_bootstrap.cc:654] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.247907  2246 tablet_bootstrap.cc:492] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: No bootstrap required, opened a new log
I20261120 03:03:02.248394  2246 ts_tablet_manager.cc:1449] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: Time spent bootstrapping tablet: real 0.043s	user 0.009s	sys 0.014s
I20261120 03:03:02.251597  2245 ts_tablet_manager.cc:1480] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.007s
I20261120 03:03:02.251835  2246 raft_consensus.cc:366] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.252869  2246 raft_consensus.cc:392] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.253268  2246 raft_consensus.cc:787] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.253899  2246 consensus_queue.cc:261] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.257562  2246 ts_tablet_manager.cc:1480] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20261120 03:03:02.260514  2246 tablet_bootstrap.cc:492] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: Bootstrap starting.
I20261120 03:03:02.264756  2245 tablet_bootstrap.cc:492] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: Bootstrap starting.
I20261120 03:03:02.272576  2246 tablet_bootstrap.cc:654] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.274636  2245 tablet_bootstrap.cc:654] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.285485  2247 tablet_bootstrap.cc:492] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: No bootstrap required, opened a new log
I20261120 03:03:02.286171  2247 ts_tablet_manager.cc:1449] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Time spent bootstrapping tablet: real 0.055s	user 0.012s	sys 0.009s
I20261120 03:03:02.286463  2246 tablet_bootstrap.cc:492] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: No bootstrap required, opened a new log
I20261120 03:03:02.287300  2246 ts_tablet_manager.cc:1449] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: Time spent bootstrapping tablet: real 0.027s	user 0.015s	sys 0.005s
I20261120 03:03:02.285481  2245 tablet_bootstrap.cc:492] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: No bootstrap required, opened a new log
I20261120 03:03:02.288419  2245 ts_tablet_manager.cc:1449] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: Time spent bootstrapping tablet: real 0.024s	user 0.007s	sys 0.010s
I20261120 03:03:02.289464  2247 raft_consensus.cc:366] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.290561  2247 raft_consensus.cc:392] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.289939  2246 raft_consensus.cc:366] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.290889  2247 raft_consensus.cc:787] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.291074  2246 raft_consensus.cc:392] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.291276  2245 raft_consensus.cc:366] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.291546  2246 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.291893  2245 raft_consensus.cc:392] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.291652  2247 consensus_queue.cc:261] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.292248  2245 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.292478  2246 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.292948  2245 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.294322  2250 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.294824  2250 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.300997  2246 ts_tablet_manager.cc:1480] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.003s
I20261120 03:03:02.301820  2252 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.302469  2252 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.318480  2250 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.319551  2250 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.320734  2250 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.321342  2252 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.323341  2252 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.324342  2252 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.323707  2250 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.341189  2246 tablet_bootstrap.cc:492] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: Bootstrap starting.
I20261120 03:03:02.357268  2247 ts_tablet_manager.cc:1480] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Time spent starting tablet: real 0.071s	user 0.016s	sys 0.006s
I20261120 03:03:02.362228  2247 tablet_bootstrap.cc:492] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Bootstrap starting.
I20261120 03:03:02.377918  2252 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.383865  2247 tablet_bootstrap.cc:654] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.385218  2246 tablet_bootstrap.cc:654] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.385598  2255 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.417294  2042 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:02.418432  2042 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 0
I20261120 03:03:02.415817  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:02.419883  2086 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters: 
I20261120 03:03:02.420099  2043 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 0
I20261120 03:03:02.420948  2259 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:03:02.421762  2259 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:02.422125  2259 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.422364  2245 ts_tablet_manager.cc:1480] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: Time spent starting tablet: real 0.134s	user 0.031s	sys 0.049s
I20261120 03:03:02.422947  2259 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.423215  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:02.397625  2255 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.425644  2198 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:02.426499  2198 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 0
I20261120 03:03:02.429916  2086 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters: 
W20261120 03:03:02.432482  2085 leader_election.cc:343] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169): Illegal state: must be running to vote when last-logged opid is not known
I20261120 03:03:02.423431  2245 tablet_bootstrap.cc:492] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: Bootstrap starting.
I20261120 03:03:02.508622  2259 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.508270  1983 catalog_manager.cc:1310] Loaded cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:02.458779  2252 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:03:02.439563  2255 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:02.546509  1972 catalog_manager.cc:1310] Loaded cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:02.608533  2245 tablet_bootstrap.cc:700] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: Time spent opening tablet: real 0.154s	user 0.003s	sys 0.000s
I20261120 03:03:02.619151  1983 catalog_manager.cc:1603] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: loading cluster ID for follower catalog manager: success
I20261120 03:03:02.620496  2252 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:02.623153  1972 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: loading cluster ID for follower catalog manager: success
I20261120 03:03:02.624706  2245 tablet_bootstrap.cc:654] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.625242  2198 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:02.633216  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:02.634094  2199 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.636853  2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:02.637665  2197 raft_consensus.cc:2628] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in current term 1: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:02.631693  2252 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.645912  2246 tablet_bootstrap.cc:492] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: No bootstrap required, opened a new log
I20261120 03:03:02.635952  2199 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 1
I20261120 03:03:02.656826  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:02.657378  2043 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.657281  2085 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 
I20261120 03:03:02.657758  2247 tablet_bootstrap.cc:492] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: No bootstrap required, opened a new log
I20261120 03:03:02.659268  2247 ts_tablet_manager.cc:1449] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Time spent bootstrapping tablet: real 0.297s	user 0.019s	sys 0.038s
I20261120 03:03:02.659410  2262 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:02.659521  2043 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 1
I20261120 03:03:02.660004  2262 raft_consensus.cc:744] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Becoming Leader. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:02.661204  2262 consensus_queue.cc:238] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.654681  2246 ts_tablet_manager.cc:1449] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: Time spent bootstrapping tablet: real 0.314s	user 0.011s	sys 0.023s
I20261120 03:03:02.667680  2266 raft_consensus.cc:534] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 764428dd82184375a0c80cbf46397e89)
I20261120 03:03:02.668306  2266 raft_consensus.cc:556] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:02.669775  2247 raft_consensus.cc:366] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.670156  2246 raft_consensus.cc:366] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.670531  2247 raft_consensus.cc:392] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.670904  2246 raft_consensus.cc:392] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.671357  2266 leader_election.cc:290] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159)
I20261120 03:03:02.671458  1849 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e68c557e0a6047f5956737cfd33fffb5" is_pre_election: true
I20261120 03:03:02.672328  1849 raft_consensus.cc:2062] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 3a4598f24f6246008d66f45cff9355af in term 1
I20261120 03:03:02.671016  2247 raft_consensus.cc:787] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.677009  1891 leader_election.cc:304] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af, e68c557e0a6047f5956737cfd33fffb5; no voters: 
I20261120 03:03:02.680531  2245 tablet_bootstrap.cc:492] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: No bootstrap required, opened a new log
I20261120 03:03:02.671308  2246 raft_consensus.cc:787] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.682585  1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:02.683976  2042 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:02.684707  2042 raft_consensus.cc:2628] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 1: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:02.686012  2246 consensus_queue.cc:261] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.688052  2246 ts_tablet_manager.cc:1480] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: Time spent starting tablet: real 0.023s	user 0.005s	sys 0.000s
I20261120 03:03:02.683529  2266 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 03:03:02.689410  2266 raft_consensus.cc:534] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 1 FOLLOWER]: Starting leader election (detected failure of leader 764428dd82184375a0c80cbf46397e89)
I20261120 03:03:02.689857  2266 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:02.690425  2245 ts_tablet_manager.cc:1449] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: Time spent bootstrapping tablet: real 0.267s	user 0.025s	sys 0.000s
I20261120 03:03:02.697743  2252 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.703401  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:02.708098  2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:02.709172  2121 raft_consensus.cc:2628] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in current term 1: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:02.709988  2245 raft_consensus.cc:366] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.703928  2043 raft_consensus.cc:2612] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader election vote request: already granted 'yes' vote to a157015bf640485fb805159465a9ab1b in term 1; re-sending same reply
I20261120 03:03:02.707741  2247 consensus_queue.cc:261] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.713672  2245 raft_consensus.cc:392] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.714031  2245 raft_consensus.cc:787] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.714797  2245 consensus_queue.cc:261] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.717041  2270 raft_consensus.cc:534] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 764428dd82184375a0c80cbf46397e89)
I20261120 03:03:02.723614  2270 raft_consensus.cc:556] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:02.728578  2256 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.728994  2256 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.729643  2247 ts_tablet_manager.cc:1480] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Time spent starting tablet: real 0.070s	user 0.009s	sys 0.001s
I20261120 03:03:02.730747  2247 tablet_bootstrap.cc:492] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Bootstrap starting.
I20261120 03:03:02.724376  1972 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: acquiring CA information for follower catalog manager: success
I20261120 03:03:02.732581  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:02.734360  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:02.734506  2259 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.731182  2252 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:02.734980  2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:02.735447  2271 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20261120 03:03:02.747435  2245 ts_tablet_manager.cc:1480] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: Time spent starting tablet: real 0.050s	user 0.008s	sys 0.000s
I20261120 03:03:02.751380  2197 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 0
I20261120 03:03:02.752305  2255 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.745491  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:02.747028  2043 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.747208  2259 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.754987  2255 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.759454  1994 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 
I20261120 03:03:02.756866  2043 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 1
I20261120 03:03:02.759476  2256 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:02.761622  1915 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af" is_pre_election: true
I20261120 03:03:02.761929  1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:02.763000  2270 leader_election.cc:290] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:02.763828  2250 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:03:02.766490  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
W20261120 03:03:02.765003  1994 leader_election.cc:343] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169): Illegal state: must be running to vote when last-logged opid is not known
I20261120 03:03:02.765813  1983 catalog_manager.cc:1625] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: acquiring CA information for follower catalog manager: success
I20261120 03:03:02.765167  2250 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:02.763335  2266 raft_consensus.cc:556] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:02.771816  2259 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:02.773003  1772 catalog_manager.cc:5856] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b reported cstate change: term changed from 0 to 1, leader changed from <none> to a157015bf640485fb805159465a9ab1b (127.0.96.66). New cstate: current_term: 1 leader_uuid: "a157015bf640485fb805159465a9ab1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:02.771870  2250 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.776104  1914 raft_consensus.cc:1303] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 FOLLOWER]: Rejecting Update request from peer 764428dd82184375a0c80cbf46397e89 for earlier term 1. Current term is 2. Ops: []
I20261120 03:03:02.771804  1825 leader_election.cc:304] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e68c557e0a6047f5956737cfd33fffb5; no voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89
I20261120 03:03:02.792665  2250 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.795655  2270 raft_consensus.cc:2966] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261120 03:03:02.798923  1996 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b
I20261120 03:03:02.800289  2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:02.806499  2255 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:02.806785  1983 catalog_manager.cc:1653] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261120 03:03:02.808326  2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:02.809010  2197 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 0
I20261120 03:03:02.810130  2247 tablet_bootstrap.cc:654] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:02.813608  2269 consensus_queue.cc:1060] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 }, Status: INVALID_TERM, Last received: 1.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20261120 03:03:02.814860  2269 raft_consensus.cc:3291] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 1 LEADER]: Stepping down as leader of term 1
I20261120 03:03:02.815279  2269 raft_consensus.cc:787] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 764428dd82184375a0c80cbf46397e89, State: Running, Role: LEADER
I20261120 03:03:02.816671  2269 consensus_queue.cc:261] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:02.817236  1996 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b
I20261120 03:03:02.817811  2269 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:02.818418  2277 raft_consensus.cc:2966] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261120 03:03:02.819567  2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:02.821072  2274 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261120 03:03:02.822641  2247 tablet_bootstrap.cc:492] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: No bootstrap required, opened a new log
I20261120 03:03:02.823276  2247 ts_tablet_manager.cc:1449] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Time spent bootstrapping tablet: real 0.093s	user 0.006s	sys 0.010s
I20261120 03:03:02.823808  2086 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters: 
I20261120 03:03:02.824765  2274 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:02.825171  2274 raft_consensus.cc:744] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Becoming Leader. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:02.825906  2274 consensus_queue.cc:238] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.827873  2258 raft_consensus.cc:2966] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20261120 03:03:02.827920  2085 leader_election.cc:343] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169): Illegal state: must be running to vote when last-logged opid is not known
I20261120 03:03:02.829823  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:02.830956  2043 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 1: Already voted for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in this term.
I20261120 03:03:02.832476  1972 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261120 03:03:02.836035  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:02.836836  2043 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a157015bf640485fb805159465a9ab1b in current term 1: Already voted for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in this term.
I20261120 03:03:02.838817  2250 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 election: Requested vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:02.840457  2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:02.841037  2197 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.841897  2197 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 1
I20261120 03:03:02.843253  2121 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 0
I20261120 03:03:02.844588  2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:02.846408  2121 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.828962  2247 raft_consensus.cc:366] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.851075  2121 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 1
I20261120 03:03:02.850909  1994 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 
I20261120 03:03:02.852660  2256 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:02.853150  2256 raft_consensus.cc:744] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 LEADER]: Becoming Leader. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:02.853250  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:02.853935  2256 consensus_queue.cc:238] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:02.852188  2247 raft_consensus.cc:392] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:02.858752  2247 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Initialized, Role: FOLLOWER
I20261120 03:03:02.862499  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:02.863821  2120 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 1: Already voted for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in this term.
I20261120 03:03:02.864691  2085 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:02.862715  2247 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.867949  2262 raft_consensus.cc:2984] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20261120 03:03:02.868577  2262 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:02.868947  2247 ts_tablet_manager.cc:1480] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Time spent starting tablet: real 0.045s	user 0.007s	sys 0.001s
I20261120 03:03:02.875973  2256 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
E20261120 03:03:02.886801  1772 catalog_manager.cc:6019] Error updating tablets from a157015bf640485fb805159465a9ab1b: Illegal state: Replica 764428dd82184375a0c80cbf46397e89 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }. Tablet report was: is_incremental: true updated_tablets { tablet_id: "ff883ba4b464463a9628eff440f1190a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "a157015bf640485fb805159465a9ab1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } sequence_number: 9
I20261120 03:03:02.891506  2256 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261120 03:03:02.898138  2165 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:02.901292  2255 raft_consensus.cc:2966] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261120 03:03:02.901666  2165 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:02.902486  2255 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261120 03:03:02.904680  2260 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:02.905138  2260 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:02.915202  2260 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:02.917703  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:02.937105  2266 leader_election.cc:290] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 2 election: Requested vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159)
I20261120 03:03:02.937829  2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:02.941951  1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89"
I20261120 03:03:02.942981  1784 raft_consensus.cc:2062] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate 3a4598f24f6246008d66f45cff9355af in term 2
I20261120 03:03:02.944809  1849 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e68c557e0a6047f5956737cfd33fffb5"
I20261120 03:03:02.948226  2197 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:02.953605  2165 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:02.954973  2255 raft_consensus.cc:2966] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20261120 03:03:02.964715  2232 master_proxy_rpc.cc:203] Re-attempting IsCreateTableDone request to leader Master (127.0.96.126:43725)
I20261120 03:03:02.971812  2274 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:02.977507  1888 leader_election.cc:304] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89; no voters: e68c557e0a6047f5956737cfd33fffb5
I20261120 03:03:02.978375  2266 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:02.978837  2266 raft_consensus.cc:744] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 LEADER]: Becoming Leader. State: Replica: 3a4598f24f6246008d66f45cff9355af, State: Running, Role: LEADER
I20261120 03:03:02.979403  2266 consensus_queue.cc:238] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:02.992273  1784 raft_consensus.cc:1338] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 2 FOLLOWER]: Refusing update from remote peer 3a4598f24f6246008d66f45cff9355af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20261120 03:03:02.994408  1849 raft_consensus.cc:3296] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:02.995379  1849 raft_consensus.cc:1338] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Refusing update from remote peer 3a4598f24f6246008d66f45cff9355af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20261120 03:03:02.994992  2292 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [LEADER]: Connected to new peer: Peer: permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20261120 03:03:03.000751  2269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a4598f24f6246008d66f45cff9355af. Latest consensus state: current_term: 2 leader_uuid: "3a4598f24f6246008d66f45cff9355af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:03.001415  2269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:03.007192  2291 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [LEADER]: Connected to new peer: Peer: permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20261120 03:03:03.008562  2275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a4598f24f6246008d66f45cff9355af. Latest consensus state: current_term: 2 leader_uuid: "3a4598f24f6246008d66f45cff9355af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:03.009217  2275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:03.012696  2266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3a4598f24f6246008d66f45cff9355af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:03.013383  2269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3a4598f24f6246008d66f45cff9355af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:03.013757  2266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:03.014021  2269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:03.025161  2295 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a4598f24f6246008d66f45cff9355af. Latest consensus state: current_term: 2 leader_uuid: "3a4598f24f6246008d66f45cff9355af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:03.025820  2295 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:03.026546  2270 raft_consensus.cc:534] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a4598f24f6246008d66f45cff9355af)
I20261120 03:03:03.027014  2294 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:03.026940  2270 raft_consensus.cc:556] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:03.030592  1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:03.031872  2297 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3a4598f24f6246008d66f45cff9355af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:03.032629  2297 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:03.033201  2270 leader_election.cc:290] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:03.034653  2294 catalog_manager.cc:694] Loaded metadata for table test-table [id=267fd8b91bac4f058896b2ba899401ba]
I20261120 03:03:03.042394  2294 tablet_loader.cc:96] loaded metadata for tablet 178ee4361c534583b29d93a1e67d3c4f (table test-table [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:03.043653  2294 tablet_loader.cc:96] loaded metadata for tablet 791003cb42cf4e50a7666946f7ffad6b (table test-table [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:03.044938  2294 tablet_loader.cc:96] loaded metadata for tablet ff883ba4b464463a9628eff440f1190a (table test-table [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:03.046310  2294 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:03.046729  1914 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af" is_pre_election: true
I20261120 03:03:03.048518  1825 leader_election.cc:304] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e68c557e0a6047f5956737cfd33fffb5; no voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89
I20261120 03:03:03.049153  2270 raft_consensus.cc:2966] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20261120 03:03:03.050232  2294 catalog_manager.cc:1310] Loaded cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:03.050529  2294 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:03.053799  2294 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:03.056787  2294 catalog_manager.cc:6248] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: Loaded TSK: 0
I20261120 03:03:03.057824  2294 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:03.080775  1905 catalog_manager.cc:5856] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b reported cstate change: term changed from 0 to 1, leader changed from <none> to a157015bf640485fb805159465a9ab1b (127.0.96.66). New cstate: current_term: 1 leader_uuid: "a157015bf640485fb805159465a9ab1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:03.087255  2291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3a4598f24f6246008d66f45cff9355af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:03.087859  2291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:03.138780  1969 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261120 03:03:03.178218  2068 heartbeater.cc:500] Master 127.0.96.124:46415 was elected leader, sending a full tablet report...
I20261120 03:03:03.182754  1905 catalog_manager.cc:5856] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 reported cstate change: term changed from 0 to 1, leader changed from <none> to a157015bf640485fb805159465a9ab1b (127.0.96.66). New cstate: current_term: 1 leader_uuid: "a157015bf640485fb805159465a9ab1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } }
I20261120 03:03:03.184288  1905 catalog_manager.cc:5856] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65). New cstate: current_term: 1 leader_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:03.199222  2146 heartbeater.cc:500] Master 127.0.96.124:46415 was elected leader, sending a full tablet report...
I20261120 03:03:03.203706  2224 heartbeater.cc:500] Master 127.0.96.124:46415 was elected leader, sending a full tablet report...
I20261120 03:03:03.426375  2042 tablet_service.cc:2056] Received LeaderStepDown RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b"
dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
 from {username='slave'} at 127.0.0.1:49178
I20261120 03:03:03.426936  2042 raft_consensus.cc:635] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 LEADER]: Received request to step down
I20261120 03:03:03.427215  2042 raft_consensus.cc:3291] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 LEADER]: Stepping down as leader of term 1
I20261120 03:03:03.427433  2042 raft_consensus.cc:787] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:03.428148  2042 consensus_queue.cc:261] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.429173  2042 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:03.595358  2259 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:03.597456  2319 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:03.595988  2259 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.597941  2319 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.599843  2319 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:03.599972  2259 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:03.600638  2042 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:03.600934  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:03.601042  2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:03.601383  2042 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 2
I20261120 03:03:03.601042  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:03.602183  2121 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 1
I20261120 03:03:03.602244  2199 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 1
I20261120 03:03:03.601655  2043 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 2
I20261120 03:03:03.602716  2086 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters: 
I20261120 03:03:03.603652  2259 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 03:03:03.604094  2259 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:03.603659  2165 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 
I20261120 03:03:03.604499  2259 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:03.605387  2259 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.606396  2319 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 03:03:03.606786  2319 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:03.607127  2319 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:03.608353  2319 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.612389  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:03.612799  2259 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 2 election: Requested vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:03.613070  2199 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a157015bf640485fb805159465a9ab1b in current term 2: Already voted for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in this term.
I20261120 03:03:03.612936  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:03.614144  2043 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 2
I20261120 03:03:03.614933  2319 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:03.615767  2042 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:03.615844  2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:03.616712  2121 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 2: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:03.616672  2042 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 2: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:03.617955  2164 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:03.618654  2319 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20261120 03:03:03.620864  2086 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c
I20261120 03:03:03.621533  2259 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:03.622031  2259 raft_consensus.cc:744] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 LEADER]: Becoming Leader. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:03.623245  2259 consensus_queue.cc:238] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.627712  2043 raft_consensus.cc:1338] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Refusing update from remote peer a157015bf640485fb805159465a9ab1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:03.627761  2199 raft_consensus.cc:1338] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Refusing update from remote peer a157015bf640485fb805159465a9ab1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:03.629319  2259 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:03.647727  1903 catalog_manager.cc:5856] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 reported cstate change: term changed from 1 to 2, leader changed from 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65) to a157015bf640485fb805159465a9ab1b (127.0.96.66). New cstate: current_term: 2 leader_uuid: "a157015bf640485fb805159465a9ab1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } }
I20261120 03:03:03.648312  2274 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:03.663630  2121 tablet_service.cc:2056] Received LeaderStepDown RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b"
dest_uuid: "a157015bf640485fb805159465a9ab1b"
 from {username='slave'} at 127.0.0.1:56228
I20261120 03:03:03.664348  2121 raft_consensus.cc:635] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 LEADER]: Received request to step down
I20261120 03:03:03.664721  2121 raft_consensus.cc:3291] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 LEADER]: Stepping down as leader of term 2
I20261120 03:03:03.665004  2121 raft_consensus.cc:787] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:03.665735  2121 consensus_queue.cc:261] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.666584  2121 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:03.868399  2320 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:03.868848  2320 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.871547  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:03.873900  2320 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:03.874303  2121 tablet_service.cc:2056] Received LeaderStepDown RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f"
dest_uuid: "a157015bf640485fb805159465a9ab1b"
 from {username='slave'} at 127.0.0.1:56228
I20261120 03:03:03.874853  2121 raft_consensus.cc:635] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Received request to step down
I20261120 03:03:03.875174  2121 raft_consensus.cc:3291] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Stepping down as leader of term 1
I20261120 03:03:03.875451  2121 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:03.876237  2121 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:03.877398  2256 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:03.877928  2256 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.879062  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:03.879755  2256 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:03.877440  2121 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:03.881052  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:03.879726  2120 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 3
I20261120 03:03:03.882232  2199 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 2
I20261120 03:03:03.883134  1994 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 
I20261120 03:03:03.883859  2256 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261120 03:03:03.884244  2256 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:03.884622  2256 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:03.884730  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:03.885442  2120 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 3
I20261120 03:03:03.885470  2256 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.888784  2165 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:03.889612  2320 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261120 03:03:03.890007  2320 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Starting leader election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:03.890380  2320 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:03.891129  2320 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.892400  2256 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 election: Requested vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:03.893208  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:03.893445  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:03.893813  2199 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in current term 3: Already voted for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in this term.
I20261120 03:03:03.894569  2120 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 3
I20261120 03:03:03.897562  2320 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 3 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:03.898211  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:03.898895  2043 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 3: Already voted for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in this term.
I20261120 03:03:03.898900  2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:03.899704  2121 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 3: Already voted for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in this term.
I20261120 03:03:03.900411  1996 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c
I20261120 03:03:03.900616  2165 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:03.901129  2256 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Leader election won for term 3
I20261120 03:03:03.901352  2320 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20261120 03:03:03.901640  2256 raft_consensus.cc:744] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 LEADER]: Becoming Leader. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:03.902495  2256 consensus_queue.cc:238] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:03.907697  2121 raft_consensus.cc:1338] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261120 03:03:03.909394  2250 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:03:03.909962  2199 raft_consensus.cc:1338] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261120 03:03:03.911548  2250 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:03:03.911646  1905 catalog_manager.cc:5856] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 reported cstate change: term changed from 2 to 3, leader changed from a157015bf640485fb805159465a9ab1b (127.0.96.66) to 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65). New cstate: current_term: 3 leader_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:03.992897  2319 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:03.993376  2319 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:03.994890  2319 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:03.995886  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:03.996368  2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:03.996980  2121 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 2
I20261120 03:03:03.997798  2165 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:03.998508  2319 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 03:03:03.998811  2319 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:03.999080  2319 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:03.999629  2319 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.004513  2319 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:04.005592  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:04.005810  2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:04.006906  2121 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 2
I20261120 03:03:04.011565  2165 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:04.012323  2319 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:04.012821  2319 raft_consensus.cc:744] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Becoming Leader. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:04.014002  2319 consensus_queue.cc:238] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.018575  2121 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:04.020144  2043 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:04.021127  2043 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:04.023406  2320 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:04.023005  1903 catalog_manager.cc:5856] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c reported cstate change: term changed from 1 to 2, leader changed from a157015bf640485fb805159465a9ab1b (127.0.96.66) to 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67). New cstate: current_term: 2 leader_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:04.031750  2320 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:04.091207  2120 tablet_service.cc:2056] Received LeaderStepDown RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a"
dest_uuid: "a157015bf640485fb805159465a9ab1b"
 from {username='slave'} at 127.0.0.1:56228
I20261120 03:03:04.091812  2120 raft_consensus.cc:635] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Received request to step down
I20261120 03:03:04.092204  2120 raft_consensus.cc:3291] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Stepping down as leader of term 1
I20261120 03:03:04.092602  2120 raft_consensus.cc:787] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:04.094293  2120 consensus_queue.cc:261] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.095515  2120 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:04.234526  2319 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:04.236404  2319 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.239189  2319 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:04.239409  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:04.240289  2043 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 1
I20261120 03:03:04.240104  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:04.240926  2120 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 2
I20261120 03:03:04.241302  2164 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 
I20261120 03:03:04.242024  2319 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 03:03:04.242267  2319 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:04.242478  2319 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:04.242990  2319 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.248483  2319 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:04.249133  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:04.249297  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:04.249583  2043 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:04.250294  2120 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 2
I20261120 03:03:04.250548  2043 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 2
I20261120 03:03:04.255501  2165 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 
I20261120 03:03:04.256215  2319 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:04.256698  2319 raft_consensus.cc:744] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Becoming Leader. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:04.257380  2319 consensus_queue.cc:238] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.261626  2120 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:04.262068  2043 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:04.263878  2320 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:04.264539  2324 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:04.267948  1905 catalog_manager.cc:5856] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c reported cstate change: term changed from 1 to 2, leader changed from a157015bf640485fb805159465a9ab1b (127.0.96.66) to 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67). New cstate: current_term: 2 leader_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:04.322803  2199 tablet_service.cc:2056] Received LeaderStepDown RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f"
dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
 from {username='slave'} at 127.0.0.1:52878
I20261120 03:03:04.323346  2199 raft_consensus.cc:635] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Received request to step down
I20261120 03:03:04.323707  2199 raft_consensus.cc:3291] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Stepping down as leader of term 2
I20261120 03:03:04.323993  2199 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:04.324757  2199 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.326014  2199 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:04.438484  2262 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 852e6c6e1ef54e5ebde6ab6f55b82b9c)
I20261120 03:03:04.438977  2262 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.440690  2262 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:04.441308  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:04.441581  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:04.442328  2199 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 3
I20261120 03:03:04.443209  2085 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:04.443809  2262 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261120 03:03:04.444207  2262 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 852e6c6e1ef54e5ebde6ab6f55b82b9c)
I20261120 03:03:04.444469  2262 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:04.445071  2262 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.451171  2262 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 3 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:04.452080  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:04.452495  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:04.453622  2199 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 3
I20261120 03:03:04.459064  2085 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:04.459654  2262 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader election won for term 3
I20261120 03:03:04.460071  2262 raft_consensus.cc:744] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 3 LEADER]: Becoming Leader. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:04.461123  2262 consensus_queue.cc:238] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.465384  2199 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Refusing update from remote peer a157015bf640485fb805159465a9ab1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261120 03:03:04.466370  2043 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:04.469219  2259 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:03:04.470098  2043 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Refusing update from remote peer a157015bf640485fb805159465a9ab1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261120 03:03:04.470942  1903 catalog_manager.cc:5856] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b reported cstate change: term changed from 2 to 3, leader changed from 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67) to a157015bf640485fb805159465a9ab1b (127.0.96.66). New cstate: current_term: 3 leader_uuid: "a157015bf640485fb805159465a9ab1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:04.487273  2259 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:03:04.530175  2199 tablet_service.cc:2056] Received LeaderStepDown RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a"
dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
 from {username='slave'} at 127.0.0.1:52878
I20261120 03:03:04.530809  2199 raft_consensus.cc:635] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Received request to step down
I20261120 03:03:04.531615  2199 raft_consensus.cc:3291] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Stepping down as leader of term 2
I20261120 03:03:04.531930  2199 raft_consensus.cc:787] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:04.532711  2199 consensus_queue.cc:261] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.534883  2199 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:04.680572  2323 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 852e6c6e1ef54e5ebde6ab6f55b82b9c)
I20261120 03:03:04.681022  2323 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.682547  2323 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:04.683568  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:04.683581  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:04.684376  2199 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 3
I20261120 03:03:04.685389  1994 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: a157015bf640485fb805159465a9ab1b
I20261120 03:03:04.686075  2323 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261120 03:03:04.686462  2323 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 852e6c6e1ef54e5ebde6ab6f55b82b9c)
I20261120 03:03:04.686719  2323 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:04.687275  2323 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.692167  2323 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 election: Requested vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:04.692848  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:04.693065  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:04.694150  2199 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 3
I20261120 03:03:04.695854  2259 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 852e6c6e1ef54e5ebde6ab6f55b82b9c)
I20261120 03:03:04.696345  2259 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.698661  2259 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:04.699772  2198 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:04.700182  1994 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: a157015bf640485fb805159465a9ab1b
I20261120 03:03:04.700533  2198 raft_consensus.cc:2628] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a157015bf640485fb805159465a9ab1b in current term 3: Already voted for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in this term.
I20261120 03:03:04.700992  2323 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Leader election won for term 3
I20261120 03:03:04.701447  2323 raft_consensus.cc:744] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 LEADER]: Becoming Leader. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:04.701792  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:04.702189  2323 consensus_queue.cc:238] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:04.703548  2086 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:04.704597  2259 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:04.705399  2259 raft_consensus.cc:2966] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20261120 03:03:04.711639  2198 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261120 03:03:04.711895  2120 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20261120 03:03:04.714844  2256 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:03:04.716208  2256 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:03:04.716796  1905 catalog_manager.cc:5856] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 reported cstate change: term changed from 2 to 3, leader changed from 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67) to 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65). New cstate: current_term: 3 leader_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:04.745097  1902 catalog_manager.cc:3588] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53956:
table { table_id: "267fd8b91bac4f058896b2ba899401ba" table_name: "test-table" } new_table_name: "NewTableName"
I20261120 03:03:04.775458  1904 catalog_manager.cc:5956] TS 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485) does not have the latest schema for tablet ff883ba4b464463a9628eff440f1190a (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba]). Expected version 1 got 0
I20261120 03:03:04.778101  1903 catalog_manager.cc:5956] TS 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169) does not have the latest schema for tablet ff883ba4b464463a9628eff440f1190a (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba]). Expected version 1 got 0
I20261120 03:03:04.780187  1905 catalog_manager.cc:5956] TS a157015bf640485fb805159465a9ab1b (127.0.96.66:36367) does not have the latest schema for tablet ff883ba4b464463a9628eff440f1190a (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba]). Expected version 1 got 0
I20261120 03:03:04.795710   385 master.cc:585] Master@127.0.96.124:46415 shutting down...
I20261120 03:03:04.829411  2336 tablet.cc:1872] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20261120 03:03:04.844023  2339 tablet.cc:1872] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20261120 03:03:04.883100  2340 tablet.cc:1872] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20261120 03:03:04.879980  2343 tablet.cc:1872] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20261120 03:03:04.907737  2341 tablet.cc:1872] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20261120 03:03:04.892259  2338 tablet.cc:1872] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20261120 03:03:04.919839  2344 tablet.cc:1872] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20261120 03:03:04.914868  2342 tablet.cc:1872] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20261120 03:03:04.952399  2339 tablet.cc:1872] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20261120 03:03:05.002301   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 LEADER]: Raft consensus shutting down.
I20261120 03:03:05.002985   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:05.003314   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: stopping tablet replica
I20261120 03:03:05.024287   385 master.cc:607] Master@127.0.96.124:46415 shutdown complete.
I20261120 03:03:05.192318  2349 raft_consensus.cc:534] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a4598f24f6246008d66f45cff9355af)
I20261120 03:03:05.192816  2349 raft_consensus.cc:556] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.194777  2349 leader_election.cc:290] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:05.195574  1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:05.196460  1784 raft_consensus.cc:2062] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 2 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate e68c557e0a6047f5956737cfd33fffb5 in term 2
I20261120 03:03:05.196489  2350 raft_consensus.cc:534] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a4598f24f6246008d66f45cff9355af)
I20261120 03:03:05.197257  2350 raft_consensus.cc:556] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.199270  2350 leader_election.cc:290] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:05.200058  1823 leader_election.cc:304] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 764428dd82184375a0c80cbf46397e89, e68c557e0a6047f5956737cfd33fffb5; no voters: 
I20261120 03:03:05.202425  2349 raft_consensus.cc:3024] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261120 03:03:05.202809  2349 raft_consensus.cc:534] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3a4598f24f6246008d66f45cff9355af)
I20261120 03:03:05.203161  2349 raft_consensus.cc:3296] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:05.208755  2349 raft_consensus.cc:556] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.213550  1849 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "764428dd82184375a0c80cbf46397e89" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "e68c557e0a6047f5956737cfd33fffb5" is_pre_election: true
I20261120 03:03:05.214264  1849 raft_consensus.cc:2628] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 764428dd82184375a0c80cbf46397e89 in current term 3: Already voted for candidate e68c557e0a6047f5956737cfd33fffb5 in this term.
W20261120 03:03:05.218288  1825 leader_election.cc:336] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415): Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111)
I20261120 03:03:05.219265  2349 leader_election.cc:290] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 election: Requested vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:05.219856  1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89"
I20261120 03:03:05.220466  1784 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:05.221333  1784 raft_consensus.cc:2062] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Leader election vote request: granting yes vote for candidate e68c557e0a6047f5956737cfd33fffb5 in term 3
W20261120 03:03:05.223701  1825 leader_election.cc:336] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415): Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111)
W20261120 03:03:05.223861  1760 leader_election.cc:336] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415): Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111)
I20261120 03:03:05.224325  1760 leader_election.cc:304] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 764428dd82184375a0c80cbf46397e89; no voters: 3a4598f24f6246008d66f45cff9355af, e68c557e0a6047f5956737cfd33fffb5
I20261120 03:03:05.225818  2356 raft_consensus.cc:2966] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20261120 03:03:05.227296  1823 leader_election.cc:304] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 764428dd82184375a0c80cbf46397e89, e68c557e0a6047f5956737cfd33fffb5; no voters: 3a4598f24f6246008d66f45cff9355af
I20261120 03:03:05.227952  2349 raft_consensus.cc:3024] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 FOLLOWER]: Leader election won for term 3
I20261120 03:03:05.228368  2349 raft_consensus.cc:744] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 LEADER]: Becoming Leader. State: Replica: e68c557e0a6047f5956737cfd33fffb5, State: Running, Role: LEADER
I20261120 03:03:05.228884  2349 consensus_queue.cc:238] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.232235  2353 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68c557e0a6047f5956737cfd33fffb5. Latest consensus state: current_term: 3 leader_uuid: "e68c557e0a6047f5956737cfd33fffb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.232895  2353 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:05.234510  2359 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:05.234413  1784 raft_consensus.cc:1338] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Refusing update from remote peer e68c557e0a6047f5956737cfd33fffb5: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20261120 03:03:05.236523   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:05.236982  1825 consensus_peers.cc:597] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 -> Peer 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415): Couldn't send request to peer 3a4598f24f6246008d66f45cff9355af. Status: Network error: Client connection negotiation failed: client connection to 127.0.96.124:46415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:05.239907  2359 catalog_manager.cc:694] Loaded metadata for table NewTableName [id=267fd8b91bac4f058896b2ba899401ba]
W20261120 03:03:05.241842  2360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:05.242283  2361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:05.241552  2353 consensus_queue.cc:1049] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20261120 03:03:05.246299  2350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68c557e0a6047f5956737cfd33fffb5. Latest consensus state: current_term: 3 leader_uuid: "e68c557e0a6047f5956737cfd33fffb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
W20261120 03:03:05.246925  2363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:05.246986  2350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:05.247373  2359 tablet_loader.cc:96] loaded metadata for tablet 178ee4361c534583b29d93a1e67d3c4f (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:05.248396   385 server_base.cc:1061] running on GCE node
I20261120 03:03:05.249442  2359 tablet_loader.cc:96] loaded metadata for tablet 791003cb42cf4e50a7666946f7ffad6b (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:05.249563   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:05.249975   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:05.250192   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143785250175 us; error 0 us; skew 500 ppm
I20261120 03:03:05.250988   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:05.250957  2359 tablet_loader.cc:96] loaded metadata for tablet ff883ba4b464463a9628eff440f1190a (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:05.252657  2359 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:05.252349  2357 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e68c557e0a6047f5956737cfd33fffb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.254088  2350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e68c557e0a6047f5956737cfd33fffb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.255016  2350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:05.254838  2357 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:05.256734   385 webserver.cc:577] Webserver started at http://127.0.96.124:36403/ using document root <none> and password file <none>
I20261120 03:03:05.257052  2359 catalog_manager.cc:1310] Loaded cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:05.257313   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:05.257325  2359 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:05.257791   385 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261120 03:03:05.261314  2359 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:05.261370   385 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20261120 03:03:05.265607  2368 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:05.265610  2359 catalog_manager.cc:6248] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: Loaded TSK: 0
I20261120 03:03:05.266644   385 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.003s
I20261120 03:03:05.266649  2359 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:05.267189   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root
uuid: "3a4598f24f6246008d66f45cff9355af"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:05.267519   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:05.295306   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:05.296468   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:05.339562   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.124:46415
I20261120 03:03:05.339727  2419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.124:46415 every 8 connection(s)
I20261120 03:03:05.347899  2420 sys_catalog.cc:263] Verifying existing consensus state
I20261120 03:03:05.353943  2420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: Bootstrap starting.
I20261120 03:03:05.387678  2224 heartbeater.cc:345] Connected to a master server at 127.0.96.124:46415
I20261120 03:03:05.408479  2225 heartbeater.cc:500] Master 127.0.96.125:41159 was elected leader, sending a full tablet report...
I20261120 03:03:05.418229  2147 heartbeater.cc:500] Master 127.0.96.125:41159 was elected leader, sending a full tablet report...
I20261120 03:03:05.421371  2146 heartbeater.cc:345] Connected to a master server at 127.0.96.124:46415
I20261120 03:03:05.422891  2069 heartbeater.cc:500] Master 127.0.96.125:41159 was elected leader, sending a full tablet report...
I20261120 03:03:05.437090  2420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=7 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 03:03:05.438274  2420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: Bootstrap complete.
I20261120 03:03:05.439838  2068 heartbeater.cc:345] Connected to a master server at 127.0.96.124:46415
I20261120 03:03:05.441468  2420 raft_consensus.cc:366] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.442163  2420 raft_consensus.cc:787] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a4598f24f6246008d66f45cff9355af, State: Initialized, Role: FOLLOWER
I20261120 03:03:05.442874  2420 consensus_queue.cc:261] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.446884  2420 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:05.447347  2395 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 2 FOLLOWER]: Advancing to term 3
I20261120 03:03:05.445328  2425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.450026  2425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:05.452018  1837 catalog_manager.cc:6409] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: NewTableName [id=267fd8b91bac4f058896b2ba899401ba] alter complete (version 1)
I20261120 03:03:05.476099  2425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e68c557e0a6047f5956737cfd33fffb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.476888  2425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:05.477728  2428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68c557e0a6047f5956737cfd33fffb5. Latest consensus state: current_term: 3 leader_uuid: "e68c557e0a6047f5956737cfd33fffb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.478266  2428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:05.480194  2358 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e68c557e0a6047f5956737cfd33fffb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.480944  2358 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:05.481473  2441 catalog_manager.cc:1310] Loaded cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:05.481712  2441 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: loading cluster ID for follower catalog manager: success
I20261120 03:03:05.486949  2441 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: acquiring CA information for follower catalog manager: success
I20261120 03:03:05.490151  2441 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261120 03:03:05.499287  2385 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" instance_seqno: 1795143781966557) as {username='slave'} at 127.0.0.1:57132; Asking this server to re-register.
I20261120 03:03:05.500416  2224 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:05.502899  2385 ts_manager.cc:210] Registered new tserver with Master: 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:05.531368  2385 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "a157015bf640485fb805159465a9ab1b" instance_seqno: 1795143781803959) as {username='slave'} at 127.0.0.1:57140; Asking this server to re-register.
I20261120 03:03:05.532611  2146 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:05.535056  2385 ts_manager.cc:210] Registered new tserver with Master: a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:05.548032  2385 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" instance_seqno: 1795143781625822) as {username='slave'} at 127.0.0.1:57154; Asking this server to re-register.
I20261120 03:03:05.549283  2068 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:05.552209  2385 ts_manager.cc:210] Registered new tserver with Master: 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:05.552872   385 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.2104213s
I20261120 03:03:05.567138   385 master.cc:585] Master@127.0.96.125:41159 shutting down...
I20261120 03:03:05.593888   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 LEADER]: Raft consensus shutting down.
I20261120 03:03:05.594985   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:05.595468   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: stopping tablet replica
I20261120 03:03:05.636190   385 master.cc:607] Master@127.0.96.125:41159 shutdown complete.
I20261120 03:03:05.682816  2428 raft_consensus.cc:534] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e68c557e0a6047f5956737cfd33fffb5)
I20261120 03:03:05.683332  2428 raft_consensus.cc:556] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.685425  2428 leader_election.cc:290] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159)
W20261120 03:03:05.691433  2372 leader_election.cc:336] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159): Network error: Client connection negotiation failed: client connection to 127.0.96.125:41159: connect: Connection refused (error 111)
I20261120 03:03:05.695572  1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:05.696635  2369 leader_election.cc:304] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af; no voters: 764428dd82184375a0c80cbf46397e89, e68c557e0a6047f5956737cfd33fffb5
I20261120 03:03:05.697260  2428 raft_consensus.cc:2966] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261120 03:03:05.774529  2447 raft_consensus.cc:534] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e68c557e0a6047f5956737cfd33fffb5)
I20261120 03:03:05.774981  2447 raft_consensus.cc:556] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.776648  2447 leader_election.cc:290] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
W20261120 03:03:05.793767  1761 leader_election.cc:336] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159): Network error: Client connection negotiation failed: client connection to 127.0.96.125:41159: connect: Connection refused (error 111)
I20261120 03:03:05.795902  2395 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "764428dd82184375a0c80cbf46397e89" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af" is_pre_election: true
I20261120 03:03:05.796558  2395 raft_consensus.cc:2062] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 764428dd82184375a0c80cbf46397e89 in term 3
I20261120 03:03:05.797650  1760 leader_election.cc:304] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89; no voters: e68c557e0a6047f5956737cfd33fffb5
I20261120 03:03:05.798246  2447 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261120 03:03:05.798602  2447 raft_consensus.cc:534] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e68c557e0a6047f5956737cfd33fffb5)
I20261120 03:03:05.798875  2447 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:05.800058  2447 raft_consensus.cc:556] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.805353  2447 leader_election.cc:290] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 4 election: Requested vote from peers e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:05.806331  2395 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "764428dd82184375a0c80cbf46397e89" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af"
I20261120 03:03:05.806816  2395 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:05.807538  2395 raft_consensus.cc:2062] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Leader election vote request: granting yes vote for candidate 764428dd82184375a0c80cbf46397e89 in term 4
W20261120 03:03:05.808777  1761 leader_election.cc:336] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159): Network error: Client connection negotiation failed: client connection to 127.0.96.125:41159: connect: Connection refused (error 111)
I20261120 03:03:05.812390  1760 leader_election.cc:304] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89; no voters: e68c557e0a6047f5956737cfd33fffb5
I20261120 03:03:05.813007  2447 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 4 FOLLOWER]: Leader election won for term 4
I20261120 03:03:05.813411  2447 raft_consensus.cc:744] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 4 LEADER]: Becoming Leader. State: Replica: 764428dd82184375a0c80cbf46397e89, State: Running, Role: LEADER
I20261120 03:03:05.814069  2447 consensus_queue.cc:238] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:05.816617  2451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764428dd82184375a0c80cbf46397e89. Latest consensus state: current_term: 4 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.817345  2451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:05.819267  2395 raft_consensus.cc:1338] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Refusing update from remote peer 764428dd82184375a0c80cbf46397e89: Log matching property violated. Preceding OpId in replica: term: 3 index: 18. Preceding OpId from leader: term: 4 index: 19. (index mismatch)
I20261120 03:03:05.820189  2454 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:05.820703  2451 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20261120 03:03:05.822624  1761 consensus_peers.cc:597] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 -> Peer e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159): Couldn't send request to peer e68c557e0a6047f5956737cfd33fffb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.96.125:41159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:05.823828  2428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764428dd82184375a0c80cbf46397e89. Latest consensus state: current_term: 4 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.824414  2428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:05.825618  2454 catalog_manager.cc:694] Loaded metadata for table NewTableName [id=267fd8b91bac4f058896b2ba899401ba]
I20261120 03:03:05.826213  2447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.826894  2447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:05.828104  2428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:05.828778  2428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:05.830032  2454 tablet_loader.cc:96] loaded metadata for tablet 178ee4361c534583b29d93a1e67d3c4f (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:05.831029  2454 tablet_loader.cc:96] loaded metadata for tablet 791003cb42cf4e50a7666946f7ffad6b (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:05.831966  2454 tablet_loader.cc:96] loaded metadata for tablet ff883ba4b464463a9628eff440f1190a (table NewTableName [id=267fd8b91bac4f058896b2ba899401ba])
I20261120 03:03:05.832973  2454 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:05.835767  2454 catalog_manager.cc:1310] Loaded cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:05.836005  2454 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:05.839247  2454 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:05.842079  2454 catalog_manager.cc:6248] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: Loaded TSK: 0
I20261120 03:03:05.842758  2454 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:05.848979   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:05.853612  2455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:05.854753  2456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:05.856671  2458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:05.857261   385 server_base.cc:1061] running on GCE node
I20261120 03:03:05.858091   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:05.858291   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:05.858445   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143785858435 us; error 0 us; skew 500 ppm
I20261120 03:03:05.858842   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:05.860872   385 webserver.cc:577] Webserver started at http://127.0.96.125:43061/ using document root <none> and password file <none>
I20261120 03:03:05.861285   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:05.861425   385 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261120 03:03:05.865442   385 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20261120 03:03:05.868069  2463 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:05.868852   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20261120 03:03:05.869194   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root
uuid: "e68c557e0a6047f5956737cfd33fffb5"
format_stamp: "Formatted at 2026-11-20 03:03:01 on dist-test-slave-54pf"
I20261120 03:03:05.869518   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:05.879878   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:05.880748   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:05.923377   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.125:41159
I20261120 03:03:05.923481  2514 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.125:41159 every 8 connection(s)
I20261120 03:03:05.931491  2515 sys_catalog.cc:263] Verifying existing consensus state
I20261120 03:03:05.935626  2515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: Bootstrap starting.
I20261120 03:03:05.975961  2070 heartbeater.cc:500] Master 127.0.96.126:43725 was elected leader, sending a full tablet report...
I20261120 03:03:05.992908  2148 heartbeater.cc:500] Master 127.0.96.126:43725 was elected leader, sending a full tablet report...
I20261120 03:03:05.993736  2226 heartbeater.cc:500] Master 127.0.96.126:43725 was elected leader, sending a full tablet report...
I20261120 03:03:05.999846  2515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=7 ignored=0} mutations{seen=14 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 03:03:06.000887  2515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: Bootstrap complete.
I20261120 03:03:06.003957  2515 raft_consensus.cc:366] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:06.004693  2515 raft_consensus.cc:787] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68c557e0a6047f5956737cfd33fffb5, State: Initialized, Role: FOLLOWER
I20261120 03:03:06.005337  2515 consensus_queue.cc:261] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:06.013132  2522 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:06.013856  2522 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:06.014706  2515 sys_catalog.cc:565] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:06.019256  2490 raft_consensus.cc:3296] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:06.025790  2069 heartbeater.cc:345] Connected to a master server at 127.0.96.125:41159
I20261120 03:03:06.033200  2225 heartbeater.cc:345] Connected to a master server at 127.0.96.125:41159
I20261120 03:03:06.036993  2147 heartbeater.cc:345] Connected to a master server at 127.0.96.125:41159
I20261120 03:03:06.048633  2527 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:06.049407  2527 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:06.049805  2522 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764428dd82184375a0c80cbf46397e89. Latest consensus state: current_term: 4 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:06.050613  2522 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [sys.catalog]: This master's current role is: FOLLOWER
I20261120 03:03:06.055310  2536 catalog_manager.cc:1310] Loaded cluster ID: d357263bb8814469bf3dac3bd0bb022c
I20261120 03:03:06.055588  2536 catalog_manager.cc:1603] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: loading cluster ID for follower catalog manager: success
I20261120 03:03:06.060144  2536 catalog_manager.cc:1625] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: acquiring CA information for follower catalog manager: success
I20261120 03:03:06.063299  2536 catalog_manager.cc:1653] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261120 03:03:06.111029  2453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "764428dd82184375a0c80cbf46397e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } } }
I20261120 03:03:06.111796  2453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:06.134670  2480 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" instance_seqno: 1795143781625822) as {username='slave'} at 127.0.0.1:37252; Asking this server to re-register.
I20261120 03:03:06.135811  2069 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:06.138926  2480 ts_manager.cc:210] Registered new tserver with Master: 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:06.140609  2479 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" instance_seqno: 1795143781966557) as {username='slave'} at 127.0.0.1:37246; Asking this server to re-register.
I20261120 03:03:06.142907  2225 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:06.144814  2479 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "a157015bf640485fb805159465a9ab1b" instance_seqno: 1795143781803959) as {username='slave'} at 127.0.0.1:37268; Asking this server to re-register.
I20261120 03:03:06.145712  2480 ts_manager.cc:210] Registered new tserver with Master: 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:06.145875  2147 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:06.148909  2480 ts_manager.cc:210] Registered new tserver with Master: a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:06.148983   385 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.222978453s
I20261120 03:03:08.421607  2521 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:08.422322  2521 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:08.425918  2521 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:08.437839  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:08.489126  2541 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:08.489840  2541 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:08.493299  2541 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
W20261120 03:03:08.496539  2086 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.96.65:40485, user_credentials={real_user=slave}} blocked reactor thread for 59894.2us
I20261120 03:03:08.499097  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:08.513994  2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:08.514659  2198 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:08.516829  2543 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:08.517311  2543 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:08.517767  2085 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:08.518551  2542 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261120 03:03:08.519375  2085 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:08.519688  2543 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:08.520341  2542 raft_consensus.cc:2966] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261120 03:03:08.522245  2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:08.533748  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:08.535211  2164 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:08.536406  2543 raft_consensus.cc:2966] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261120 03:03:08.571172  2537 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:08.571661  2537 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:08.573315  2537 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:08.573787  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:08.574211  2198 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:08.575402  1994 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b
I20261120 03:03:08.576069  2537 raft_consensus.cc:2966] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20261120 03:03:09.427834  2064 debug-util.cc:398] Leaking SignalData structure 0x7b08000fe7c0 after lost signal to thread 388
W20261120 03:03:09.429066  2064 debug-util.cc:398] Leaking SignalData structure 0x7b0800125560 after lost signal to thread 1808
W20261120 03:03:09.430126  2064 debug-util.cc:398] Leaking SignalData structure 0x7b08001190c0 after lost signal to thread 2067
W20261120 03:03:09.431149  2064 debug-util.cc:398] Leaking SignalData structure 0x7b08000c9220 after lost signal to thread 2145
W20261120 03:03:09.432082  2064 debug-util.cc:398] Leaking SignalData structure 0x7b080017a0c0 after lost signal to thread 2223
W20261120 03:03:09.433106  2064 debug-util.cc:398] Leaking SignalData structure 0x7b08000306a0 after lost signal to thread 2419
W20261120 03:03:09.434125  2064 debug-util.cc:398] Leaking SignalData structure 0x7b080002e8e0 after lost signal to thread 2514
I20261120 03:03:25.919606  2570 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:25.920817  2570 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:25.966727  2570 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:25.985739  2566 raft_consensus.cc:534] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 764428dd82184375a0c80cbf46397e89)
I20261120 03:03:25.986351  2566 raft_consensus.cc:556] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:25.993346  2566 leader_election.cc:290] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159)
I20261120 03:03:25.990624  2567 raft_consensus.cc:534] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 764428dd82184375a0c80cbf46397e89)
I20261120 03:03:25.995062  2567 raft_consensus.cc:556] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:25.998967  2567 leader_election.cc:290] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:26.001551  2572 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:26.002071  2572 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.004706  2573 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:26.005141  2573 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.005653  2572 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:26.007097  2573 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:26.024778  2576 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:26.025405  2576 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:26.027194  2198 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:26.028138  2198 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 3
I20261120 03:03:26.028476  2576 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:26.029687  1994 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 
I20261120 03:03:26.031165  2570 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261120 03:03:26.031587  2570 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:26.031955  2570 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Advancing to term 4
W20261120 03:03:26.032454  2467 thread.cc:642] raft [worker] (thread pool) Time spent creating pthread: real 0.980s	user 0.000s	sys 0.016s
I20261120 03:03:26.032873  2570 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
W20261120 03:03:26.034677  2467 thread.cc:609] raft [worker] (thread pool) Time spent starting thread: real 0.983s	user 0.000s	sys 0.017s
I20261120 03:03:26.038396  2576 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:26.038846  2576 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.043550  2576 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:26.047367  2570 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 4 election: Requested vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:26.049599  2198 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:26.050231  2198 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:26.051043  2198 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 4
I20261120 03:03:26.059181  1994 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 
I20261120 03:03:26.059965  2570 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader election won for term 4
I20261120 03:03:26.060451  2570 raft_consensus.cc:744] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 LEADER]: Becoming Leader. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:26.061314  2570 consensus_queue.cc:238] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.064882  2121 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:26.065409  2198 raft_consensus.cc:1303] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Rejecting Update request from peer a157015bf640485fb805159465a9ab1b for earlier term 3. Current term is 4. Ops: []
I20261120 03:03:26.065567  2121 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 3
I20261120 03:03:26.066212  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:26.066807  2576 consensus_queue.cc:1060] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20261120 03:03:26.067619  2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:26.067827  2576 raft_consensus.cc:3291] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 3 LEADER]: Stepping down as leader of term 3
I20261120 03:03:26.069126  2118 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:26.071591  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:26.068394  2576 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:26.074138  2117 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:26.078526  2042 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:26.079425  2041 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:26.081430  2576 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.086398  2040 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:26.090018  2043 raft_consensus.cc:1303] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 LEADER]: Rejecting Update request from peer a157015bf640485fb805159465a9ab1b for earlier term 3. Current term is 4. Ops: []
I20261120 03:03:26.089625  2576 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:26.094553  2086 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:26.095289  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:26.095645  2086 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:26.099583  2574 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261120 03:03:26.100436  2574 raft_consensus.cc:2966] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261120 03:03:26.101238  2165 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 
I20261120 03:03:26.103813  2573 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261120 03:03:26.104112  2573 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:26.104579  2573 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:26.105736  2573 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.109328  2198 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20261120 03:03:26.114035  2371 thread.cc:642] raft [worker] (thread pool) Time spent creating pthread: real 1.078s	user 0.004s	sys 0.025s
W20261120 03:03:26.114409  2371 thread.cc:609] raft [worker] (thread pool) Time spent starting thread: real 1.079s	user 0.004s	sys 0.025s
I20261120 03:03:26.116346  2573 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:26.122375  2577 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20261120 03:03:26.124555  2041 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:26.135413  1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 5 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:26.153277  2165 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:26.154888  2119 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:26.156370  2573 raft_consensus.cc:2966] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261120 03:03:26.158819  2165 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:26.159762  2573 raft_consensus.cc:2966] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20261120 03:03:26.162674  2197 raft_consensus.cc:1303] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Rejecting Update request from peer 9ac1cc0b02fd4724b4680da6a7e2a875 for earlier term 3. Current term is 4. Ops: []
I20261120 03:03:26.139104  1771 catalog_manager.cc:5856] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 reported cstate change: term changed from 3 to 4, leader changed from a157015bf640485fb805159465a9ab1b (127.0.96.66) to 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65). New cstate: current_term: 4 leader_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:26.164039  2577 consensus_queue.cc:1060] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20261120 03:03:26.164544  2117 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Leader election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 4
I20261120 03:03:26.164916  2577 raft_consensus.cc:3291] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 LEADER]: Stepping down as leader of term 3
I20261120 03:03:26.165241  2577 raft_consensus.cc:787] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:26.165911  1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 5 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:26.165952  2120 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20261120 03:03:26.165961  2577 consensus_queue.cc:261] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.170043  2577 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:26.172508  2395 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 5 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af" is_pre_election: true
I20261120 03:03:26.174294  2490 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 5 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: false dest_uuid: "e68c557e0a6047f5956737cfd33fffb5" is_pre_election: true
I20261120 03:03:26.174866  2466 leader_election.cc:304] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e68c557e0a6047f5956737cfd33fffb5; no voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89
I20261120 03:03:26.176299  2567 raft_consensus.cc:2966] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261120 03:03:26.178485  2372 leader_election.cc:304] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af; no voters: 764428dd82184375a0c80cbf46397e89, e68c557e0a6047f5956737cfd33fffb5
I20261120 03:03:26.179473  2566 raft_consensus.cc:2966] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261120 03:03:26.183800  2568 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20261120 03:03:26.351231  2582 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:26.351682  2582 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.353302  2582 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:26.353926  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:26.354166  2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:26.354449  2043 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 4
I20261120 03:03:26.354815  2197 raft_consensus.cc:2628] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a157015bf640485fb805159465a9ab1b in current term 4: Already voted for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in this term.
I20261120 03:03:26.355242  2086 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters: 
I20261120 03:03:26.355885  2582 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261120 03:03:26.356186  2582 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:26.356427  2582 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:26.357017  2582 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.362284  2582 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:26.362972  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:26.363145  2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:26.363799  2197 raft_consensus.cc:2628] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a157015bf640485fb805159465a9ab1b in current term 4: Already voted for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in this term.
I20261120 03:03:26.363834  2043 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 4
I20261120 03:03:26.371343  2086 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c
I20261120 03:03:26.372053  2582 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Leader election won for term 4
I20261120 03:03:26.372620  2582 raft_consensus.cc:744] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 4 LEADER]: Becoming Leader. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:26.373708  2582 consensus_queue.cc:238] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:26.379702  2197 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Refusing update from remote peer a157015bf640485fb805159465a9ab1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20261120 03:03:26.380620  2043 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Refusing update from remote peer a157015bf640485fb805159465a9ab1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20261120 03:03:26.381104  2582 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20261120 03:03:26.382413  2582 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20261120 03:03:26.397804  1771 catalog_manager.cc:5856] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b reported cstate change: term changed from 3 to 4, leader changed from 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65) to a157015bf640485fb805159465a9ab1b (127.0.96.66). New cstate: current_term: 4 leader_uuid: "a157015bf640485fb805159465a9ab1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } }
W20261120 03:03:26.966791   390 debug-util.cc:398] Leaking SignalData structure 0x7b080002c2a0 after lost signal to thread 2371
W20261120 03:03:27.104789  2511 debug-util.cc:398] Leaking SignalData structure 0x7b08000e41c0 after lost signal to thread 388
W20261120 03:03:27.105719  2511 debug-util.cc:398] Leaking SignalData structure 0x7b0800172b60 after lost signal to thread 1808
W20261120 03:03:27.106745  2511 debug-util.cc:398] Leaking SignalData structure 0x7b08001809c0 after lost signal to thread 2067
W20261120 03:03:27.107558  2511 debug-util.cc:398] Leaking SignalData structure 0x7b08000c5fa0 after lost signal to thread 2145
W20261120 03:03:27.108366  2511 debug-util.cc:398] Leaking SignalData structure 0x7b0800052220 after lost signal to thread 2223
W20261120 03:03:27.109175  2511 debug-util.cc:398] Leaking SignalData structure 0x7b080016adc0 after lost signal to thread 2419
W20261120 03:03:27.109957  2511 debug-util.cc:398] Leaking SignalData structure 0x7b0800059280 after lost signal to thread 2514
W20261120 03:03:29.078586  2064 debug-util.cc:398] Leaking SignalData structure 0x7b080002eb40 after lost signal to thread 388
I20261120 03:03:29.170542  2608 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:29.171120  2608 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
W20261120 03:03:29.172451  1994 thread.cc:642] raft [worker] (thread pool) Time spent creating pthread: real 0.918s	user 0.011s	sys 0.012s
W20261120 03:03:29.172875  1994 thread.cc:609] raft [worker] (thread pool) Time spent starting thread: real 0.919s	user 0.011s	sys 0.012s
I20261120 03:03:29.176059  2608 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:29.215683  2479 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37246 (request call id 206) took 1156 ms. Trace:
I20261120 03:03:29.216790  2479 rpcz_store.cc:276] 1120 03:03:28.058935 (+     0us) service_pool.cc:167] Inserting onto call queue
1120 03:03:28.075291 (+ 16356us) service_pool.cc:224] Handling call
1120 03:03:29.215262 (+1139971us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20261120 03:03:29.470950  1994 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.96.67:35169 timed out after 0.263s (SENT) [suppressed 18 similar messages]
W20261120 03:03:29.763438  1994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.96.67:35169, user_credentials={real_user=slave}} blocked reactor thread for 292647us
I20261120 03:03:29.954772  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
W20261120 03:03:30.067462  1994 leader_election.cc:336] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169): Timed out: RequestConsensusVote RPC to 127.0.96.67:35169 timed out after 0.263s (SENT)
I20261120 03:03:30.082844  2199 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 4
W20261120 03:03:30.083700  2199 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52464 (request call id 487) took 279 ms (client timeout 263 ms). Trace:
W20261120 03:03:30.084311  2199 rpcz_store.cc:269] 1120 03:03:29.804667 (+     0us) service_pool.cc:167] Inserting onto call queue
1120 03:03:29.954020 (+149353us) service_pool.cc:224] Handling call
1120 03:03:30.083549 (+129529us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20261120 03:03:30.644570  2611 raft_consensus.cc:534] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 764428dd82184375a0c80cbf46397e89)
I20261120 03:03:30.645686  2611 raft_consensus.cc:556] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:30.649366  2611 leader_election.cc:290] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), e68c557e0a6047f5956737cfd33fffb5 (127.0.96.125:41159)
I20261120 03:03:30.650527  1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:30.667115  2613 raft_consensus.cc:534] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 764428dd82184375a0c80cbf46397e89)
I20261120 03:03:30.667702  2613 raft_consensus.cc:556] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "764428dd82184375a0c80cbf46397e89" member_type: VOTER last_known_addr { host: "127.0.96.126" port: 43725 } } peers { permanent_uuid: "e68c557e0a6047f5956737cfd33fffb5" member_type: VOTER last_known_addr { host: "127.0.96.125" port: 41159 } } peers { permanent_uuid: "3a4598f24f6246008d66f45cff9355af" member_type: VOTER last_known_addr { host: "127.0.96.124" port: 46415 } }
I20261120 03:03:30.670565  2613 leader_election.cc:290] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 764428dd82184375a0c80cbf46397e89 (127.0.96.126:43725), 3a4598f24f6246008d66f45cff9355af (127.0.96.124:46415)
I20261120 03:03:30.685091  2614 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:30.685746  2614 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:30.688694  2614 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:30.974197  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:30.975368  2199 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 4
W20261120 03:03:31.069312  2466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.96.124:46415, user_credentials={real_user=slave}} blocked reactor thread for 144242us
W20261120 03:03:31.070713  2085 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.96.67:35169, user_credentials={real_user=slave}} blocked reactor thread for 116407us
I20261120 03:03:31.074254  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:31.075187  2199 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 4
I20261120 03:03:31.137580  2197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52464 (request call id 486) took 1909 ms. Trace:
I20261120 03:03:31.138012  2197 rpcz_store.cc:276] 1120 03:03:29.228230 (+     0us) service_pool.cc:167] Inserting onto call queue
1120 03:03:31.136437 (+1908207us) service_pool.cc:224] Handling call
1120 03:03:31.136815 (+   378us) raft_consensus.cc:1536] Updating replica for 0 ops
1120 03:03:31.137062 (+   247us) raft_consensus.cc:1602] Early marking committed up to index 0
1120 03:03:31.137111 (+    49us) raft_consensus.cc:1607] Triggering prepare for 0 ops
1120 03:03:31.137164 (+    53us) raft_consensus.cc:1713] Marking committed up to 6
1120 03:03:31.137201 (+    37us) raft_consensus.cc:1765] Filling consensus response to leader.
1120 03:03:31.137279 (+    78us) raft_consensus.cc:1759] UpdateReplicas() finished
1120 03:03:31.137433 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20261120 03:03:31.139463  1784 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: false dest_uuid: "764428dd82184375a0c80cbf46397e89" is_pre_election: true
I20261120 03:03:31.142298  2085 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 
W20261120 03:03:31.142944  2198 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52464 (request call id 485) took 1915 ms (client timeout 263 ms). Trace:
I20261120 03:03:31.145078  2614 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
W20261120 03:03:31.143386  2198 rpcz_store.cc:269] 1120 03:03:29.227776 (+     0us) service_pool.cc:167] Inserting onto call queue
1120 03:03:31.142633 (+1914857us) service_pool.cc:211] Skipping call since client already timed out
1120 03:03:31.142758 (+   125us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20261120 03:03:31.146209  2614 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:31.148711  2618 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:31.152758  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:31.150465  2618 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.154177  2199 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 3
I20261120 03:03:31.156332  2085 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b; no voters: 
W20261120 03:03:31.157568  1996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.96.66:36367, user_credentials={real_user=slave}} blocked reactor thread for 225186us
I20261120 03:03:31.160411  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.161027  2043 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 4
I20261120 03:03:31.162818  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.164433  2164 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 
I20261120 03:03:31.156929  2618 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:31.168243  2619 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20261120 03:03:31.170053  2118 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:31.171126  2614 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:31.174387  1996 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, a157015bf640485fb805159465a9ab1b
W20261120 03:03:31.175002  2064 debug-util.cc:398] Leaking SignalData structure 0x7b0800173d60 after lost signal to thread 1808
I20261120 03:03:31.167487  2618 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:31.176203  2618 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
W20261120 03:03:31.176750  2064 debug-util.cc:398] Leaking SignalData structure 0x7b080012aee0 after lost signal to thread 2067
I20261120 03:03:31.177482  2620 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20261120 03:03:31.177868  2620 raft_consensus.cc:534] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting leader election (detected failure of leader a157015bf640485fb805159465a9ab1b)
I20261120 03:03:31.178220  2618 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:31.178995  2614 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261120 03:03:31.179212  2620 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Advancing to term 5
I20261120 03:03:31.179543  2614 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:31.180032  2614 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:31.171741  2619 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:31.183588  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:31.182703  2619 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Advancing to term 5
I20261120 03:03:31.180931  2614 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:31.189023  2118 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:31.189772  2584 raft_consensus.cc:2966] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261120 03:03:31.190753  2117 raft_consensus.cc:1303] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Rejecting Update request from peer 9ac1cc0b02fd4724b4680da6a7e2a875 for earlier term 3. Current term is 4. Ops: []
I20261120 03:03:31.196256  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:31.197242  2395 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af" is_pre_election: true
I20261120 03:03:31.205560  2490 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a4598f24f6246008d66f45cff9355af" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: false dest_uuid: "e68c557e0a6047f5956737cfd33fffb5" is_pre_election: true
I20261120 03:03:31.206918  2394 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e68c557e0a6047f5956737cfd33fffb5" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: false dest_uuid: "3a4598f24f6246008d66f45cff9355af" is_pre_election: true
I20261120 03:03:31.209281  2372 leader_election.cc:304] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a4598f24f6246008d66f45cff9355af; no voters: 764428dd82184375a0c80cbf46397e89, e68c557e0a6047f5956737cfd33fffb5
I20261120 03:03:31.210539  2466 leader_election.cc:304] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e68c557e0a6047f5956737cfd33fffb5; no voters: 3a4598f24f6246008d66f45cff9355af, 764428dd82184375a0c80cbf46397e89
I20261120 03:03:31.204623  2620 raft_consensus.cc:556] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
W20261120 03:03:31.218573  2064 debug-util.cc:398] Leaking SignalData structure 0x7b08000a04e0 after lost signal to thread 2145
I20261120 03:03:31.222122  2620 leader_election.cc:290] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
W20261120 03:03:31.222885  2164 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.96.65:40485, user_credentials={real_user=slave}} blocked reactor thread for 58607.2us
I20261120 03:03:31.224704  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.226413  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:31.226778  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ff883ba4b464463a9628eff440f1190a" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:31.227283  2043 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Advancing to term 5
I20261120 03:03:31.228353  2043 raft_consensus.cc:2062] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Leader election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 5
I20261120 03:03:31.229494  2164 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c; no voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b
I20261120 03:03:31.236379  2620 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20261120 03:03:31.241268  2064 debug-util.cc:398] Leaking SignalData structure 0x7b0800239e40 after lost signal to thread 2223
W20261120 03:03:31.242628  2064 debug-util.cc:398] Leaking SignalData structure 0x7b080023b6a0 after lost signal to thread 2419
I20261120 03:03:31.246129  2607 consensus_queue.cc:1060] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20261120 03:03:31.248309  2064 debug-util.cc:398] Leaking SignalData structure 0x7b08000a9000 after lost signal to thread 2514
I20261120 03:03:31.256822  2619 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
W20261120 03:03:31.268473  2372 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.96.125:41159, user_credentials={real_user=slave}} blocked reactor thread for 59392.1us
I20261120 03:03:31.271310  2626 raft_consensus.cc:2966] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261120 03:03:31.278391  2197 raft_consensus.cc:1303] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Rejecting Update request from peer a157015bf640485fb805159465a9ab1b for earlier term 4. Current term is 5. Ops: []
I20261120 03:03:31.280175  2624 consensus_queue.cc:1060] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20261120 03:03:31.282713  2614 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 election: Requested vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:31.283425  2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
W20261120 03:03:31.288208  2086 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.96.65:40485, user_credentials={real_user=slave}} blocked reactor thread for 58205.5us
I20261120 03:03:31.309310  2619 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 5 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:31.310484  2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:31.314849  2164 leader_election.cc:304] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: a157015bf640485fb805159465a9ab1b
I20261120 03:03:31.315654  2625 raft_consensus.cc:2966] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261120 03:03:31.315944  2623 raft_consensus.cc:3024] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Leader election won for term 5
I20261120 03:03:31.316617  2623 raft_consensus.cc:744] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 LEADER]: Becoming Leader. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:31.318415  2623 consensus_queue.cc:238] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.320418  2584 raft_consensus.cc:3291] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 LEADER]: Stepping down as leader of term 3
I20261120 03:03:31.320755  2584 raft_consensus.cc:787] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:31.322912  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.321527  2584 consensus_queue.cc:261] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:31.324043  2041 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:31.324589  2040 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:31.327000  2117 raft_consensus.cc:1303] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Rejecting Update request from peer 9ac1cc0b02fd4724b4680da6a7e2a875 for earlier term 4. Current term is 5. Ops: []
I20261120 03:03:31.327385  2086 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:31.330685  2614 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20261120 03:03:31.332157  2621 raft_consensus.cc:3291] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 4 LEADER]: Stepping down as leader of term 4
I20261120 03:03:31.335189  2621 raft_consensus.cc:787] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a157015bf640485fb805159465a9ab1b, State: Running, Role: LEADER
I20261120 03:03:31.336385  2621 consensus_queue.cc:261] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.328559  2086 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:31.340649  2614 raft_consensus.cc:2966] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20261120 03:03:31.337630  2627 consensus_queue.cc:1060] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20261120 03:03:31.334273  2584 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:31.342845  2628 raft_consensus.cc:3291] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 LEADER]: Stepping down as leader of term 4
I20261120 03:03:31.343246  2628 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:31.344440  2041 raft_consensus.cc:1303] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Rejecting Update request from peer a157015bf640485fb805159465a9ab1b for earlier term 4. Current term is 5. Ops: []
I20261120 03:03:31.344277  2628 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.345388  2628 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Advancing to term 5
I20261120 03:03:31.346640  2040 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20261120 03:03:31.338574  2621 raft_consensus.cc:3296] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Advancing to term 5
I20261120 03:03:31.358151  2043 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 4
I20261120 03:03:31.358773  2620 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20261120 03:03:31.351503  1773 catalog_manager.cc:5856] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c reported cstate change: term changed from 4 to 5, leader changed from a157015bf640485fb805159465a9ab1b (127.0.96.66) to 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67). New cstate: current_term: 5 leader_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:31.364535  2117 raft_consensus.cc:1338] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20261120 03:03:31.366616  2620 consensus_queue.cc:1049] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20261120 03:03:31.428067  2623 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:31.428699  2623 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.447011  2623 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:31.450060  2040 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.454062  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:31.454815  2120 raft_consensus.cc:2628] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 5: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:31.456002  2040 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 5
I20261120 03:03:31.459084  2164 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: a157015bf640485fb805159465a9ab1b
I20261120 03:03:31.460421  2620 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20261120 03:03:31.460778  2620 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:31.461153  2620 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Advancing to term 5
I20261120 03:03:31.461959  2620 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.460036  2623 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:31.463570  2623 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:31.467921  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:31.467929  2040 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.468798  2040 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 4
I20261120 03:03:31.468806  2120 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 4: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:31.470304  2164 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 
I20261120 03:03:31.470265  2623 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:31.483491  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:31.486826  2620 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:31.488404  2618 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261120 03:03:31.498382  2120 raft_consensus.cc:2628] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 5: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:31.494966  2618 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:31.501966  2618 raft_consensus.cc:3296] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 3 FOLLOWER]: Advancing to term 4
I20261120 03:03:31.503022  2618 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
W20261120 03:03:31.514716  2164 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.96.65:40485, user_credentials={real_user=slave}} blocked reactor thread for 44472.6us
I20261120 03:03:31.515975  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:31.517017  2043 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Leader election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 5
I20261120 03:03:31.517405  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:31.518385  2120 raft_consensus.cc:2628] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in current term 4: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:31.519582  2618 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), a157015bf640485fb805159465a9ab1b (127.0.96.66:36367)
I20261120 03:03:31.525252  2040 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:31.527350  2040 raft_consensus.cc:2062] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Leader election vote request: granting yes vote for candidate 852e6c6e1ef54e5ebde6ab6f55b82b9c in term 4
I20261120 03:03:31.529824  2164 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: a157015bf640485fb805159465a9ab1b
I20261120 03:03:31.532694  2620 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Leader election won for term 5
I20261120 03:03:31.533797  2620 raft_consensus.cc:744] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 LEADER]: Becoming Leader. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:31.542305  2164 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: a157015bf640485fb805159465a9ab1b
I20261120 03:03:31.543284  2618 raft_consensus.cc:3024] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Leader election won for term 4
I20261120 03:03:31.543023  2620 consensus_queue.cc:238] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.543918  2618 raft_consensus.cc:744] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 LEADER]: Becoming Leader. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:31.545071  2618 consensus_queue.cc:238] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:31.554735  2120 raft_consensus.cc:1338] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20261120 03:03:31.556659  2643 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20261120 03:03:31.560480  2040 raft_consensus.cc:1338] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20261120 03:03:31.562053  2643 consensus_queue.cc:1049] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20261120 03:03:31.572944  2621 raft_consensus.cc:534] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:31.573529  2624 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:31.573433  2621 raft_consensus.cc:556] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } }
I20261120 03:03:31.577041  2624 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.578545  2621 leader_election.cc:290] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169), 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485)
I20261120 03:03:31.578908  2624 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:31.581111  2043 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.581811  2197 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:31.582156  2041 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "791003cb42cf4e50a7666946f7ffad6b" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" is_pre_election: true
I20261120 03:03:31.582167  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:31.583480  2086 leader_election.cc:304] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:31.581825  2043 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate a157015bf640485fb805159465a9ab1b in term 5
I20261120 03:03:31.585072  2086 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac1cc0b02fd4724b4680da6a7e2a875, a157015bf640485fb805159465a9ab1b; no voters: 
I20261120 03:03:31.590294  2043 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20261120 03:03:31.599601  2620 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20261120 03:03:31.599571  1771 catalog_manager.cc:5856] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c reported cstate change: term changed from 4 to 5, leader changed from 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65) to 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67). New cstate: current_term: 5 leader_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:31.586694  2624 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20261120 03:03:31.609215  2117 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Refusing update from remote peer 852e6c6e1ef54e5ebde6ab6f55b82b9c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20261120 03:03:31.609843  2624 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:31.610425  2624 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Advancing to term 6
I20261120 03:03:31.611833  2624 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.613445  1771 catalog_manager.cc:5856] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c reported cstate change: term changed from 3 to 4, leader changed from 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65) to 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67). New cstate: current_term: 4 leader_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:31.614897  2618 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20261120 03:03:31.618289  2117 raft_consensus.cc:1303] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Rejecting Update request from peer 852e6c6e1ef54e5ebde6ab6f55b82b9c for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20261120 03:03:31.626492  2621 raft_consensus.cc:2966] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261120 03:03:31.630473  2618 consensus_queue.cc:1060] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20261120 03:03:31.658401  2624 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 6 election: Requested vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:31.659304  2643 raft_consensus.cc:3291] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 LEADER]: Stepping down as leader of term 5
I20261120 03:03:31.659648  2643 raft_consensus.cc:787] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 852e6c6e1ef54e5ebde6ab6f55b82b9c, State: Running, Role: LEADER
I20261120 03:03:31.659953  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:31.660420  2643 consensus_queue.cc:261] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.661688  2040 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875"
I20261120 03:03:31.663844  2643 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Advancing to term 6
I20261120 03:03:31.671542  2199 raft_consensus.cc:2044] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 6 FOLLOWER]: Leader election vote request: denying vote to candidate a157015bf640485fb805159465a9ab1b for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than candidate's term: 4 index: 6
I20261120 03:03:31.672935  2085 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
I20261120 03:03:31.673687  2614 raft_consensus.cc:2966] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20261120 03:03:31.803313  2628 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 852e6c6e1ef54e5ebde6ab6f55b82b9c)
I20261120 03:03:31.803930  2628 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.805915  2628 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:31.806906  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:31.807582  2199 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 6 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 6
I20261120 03:03:31.808615  1994 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: 
I20261120 03:03:31.809295  2628 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20261120 03:03:31.809687  2628 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 852e6c6e1ef54e5ebde6ab6f55b82b9c)
I20261120 03:03:31.810071  2628 raft_consensus.cc:3296] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Advancing to term 6
I20261120 03:03:31.810030  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b" is_pre_election: true
I20261120 03:03:31.811012  2120 raft_consensus.cc:2628] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in current term 6: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:31.811048  2628 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.826243  2628 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 6 election: Requested vote from peers a157015bf640485fb805159465a9ab1b (127.0.96.66:36367), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:31.827225  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c"
I20261120 03:03:31.827701  2120 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a157015bf640485fb805159465a9ab1b"
I20261120 03:03:31.828267  2199 raft_consensus.cc:2062] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 6 FOLLOWER]: Leader election vote request: granting yes vote for candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in term 6
I20261120 03:03:31.828418  2120 raft_consensus.cc:2628] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9ac1cc0b02fd4724b4680da6a7e2a875 in current term 6: Already voted for candidate a157015bf640485fb805159465a9ab1b in this term.
I20261120 03:03:31.835383  1994 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875; no voters: a157015bf640485fb805159465a9ab1b
I20261120 03:03:31.836467  2607 raft_consensus.cc:3024] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 6 FOLLOWER]: Leader election won for term 6
I20261120 03:03:31.837337  2607 raft_consensus.cc:744] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 6 LEADER]: Becoming Leader. State: Replica: 9ac1cc0b02fd4724b4680da6a7e2a875, State: Running, Role: LEADER
I20261120 03:03:31.838129  2607 consensus_queue.cc:238] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:31.843328  2199 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 6 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20261120 03:03:31.843664  2120 raft_consensus.cc:1338] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Refusing update from remote peer 9ac1cc0b02fd4724b4680da6a7e2a875: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20261120 03:03:31.844969  2628 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20261120 03:03:31.847951  1772 catalog_manager.cc:5856] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 reported cstate change: term changed from 5 to 6, leader changed from 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67) to 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65). New cstate: current_term: 6 leader_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:31.851580  2628 consensus_queue.cc:1049] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:784: Failure
Expected equality of these values:
  live_row_count
    Which is: 67
  table_info->GetMetrics()->live_row_count->value()
    Which is: 0
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:402: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:768: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
  Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:786: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 784, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
  Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:896: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20261120 03:03:36.371076   385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:36.399328   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:36.399921   385 tablet_replica.cc:333] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875: stopping tablet replica
I20261120 03:03:36.400439   385 raft_consensus.cc:2440] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.400847   385 raft_consensus.cc:2469] T 791003cb42cf4e50a7666946f7ffad6b P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.403102   385 tablet_replica.cc:333] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875: stopping tablet replica
W20261120 03:03:36.403208  2164 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
I20261120 03:03:36.403661   385 raft_consensus.cc:2440] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 6 LEADER]: Raft consensus shutting down.
I20261120 03:03:36.404613   385 raft_consensus.cc:2469] T 178ee4361c534583b29d93a1e67d3c4f P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 6 FOLLOWER]: Raft consensus is shut down!
W20261120 03:03:36.405910  2164 consensus_peers.cc:597] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c -> Peer 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485): Couldn't send request to peer 9ac1cc0b02fd4724b4680da6a7e2a875. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:36.406553   385 tablet_replica.cc:333] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875: stopping tablet replica
I20261120 03:03:36.406987   385 raft_consensus.cc:2440] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.407342   385 raft_consensus.cc:2469] T ff883ba4b464463a9628eff440f1190a P 9ac1cc0b02fd4724b4680da6a7e2a875 [term 5 FOLLOWER]: Raft consensus is shut down!
W20261120 03:03:36.435228  2164 consensus_peers.cc:597] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c -> Peer 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485): Couldn't send request to peer 9ac1cc0b02fd4724b4680da6a7e2a875. Status: Network error: Client connection negotiation failed: client connection to 127.0.96.65:40485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:36.461351   385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:36.475200   385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:03:36.504944  2664 raft_consensus.cc:534] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9ac1cc0b02fd4724b4680da6a7e2a875)
I20261120 03:03:36.505401  2664 raft_consensus.cc:556] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ac1cc0b02fd4724b4680da6a7e2a875" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40485 } } peers { permanent_uuid: "a157015bf640485fb805159465a9ab1b" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 36367 } } peers { permanent_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 35169 } }
I20261120 03:03:36.506110   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:36.506793   385 tablet_replica.cc:333] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b: stopping tablet replica
I20261120 03:03:36.507069  2664 leader_election.cc:290] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485), 852e6c6e1ef54e5ebde6ab6f55b82b9c (127.0.96.67:35169)
I20261120 03:03:36.507611   385 raft_consensus.cc:2440] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.508332   385 raft_consensus.cc:2469] T 791003cb42cf4e50a7666946f7ffad6b P a157015bf640485fb805159465a9ab1b [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.509191  2199 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "178ee4361c534583b29d93a1e67d3c4f" candidate_uuid: "a157015bf640485fb805159465a9ab1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "852e6c6e1ef54e5ebde6ab6f55b82b9c" is_pre_election: true
I20261120 03:03:36.510725   385 tablet_replica.cc:333] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: stopping tablet replica
I20261120 03:03:36.511359   385 raft_consensus.cc:2440] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.511859   385 raft_consensus.cc:2469] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [term 6 FOLLOWER]: Raft consensus is shut down!
W20261120 03:03:36.512672  2086 leader_election.cc:336] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9ac1cc0b02fd4724b4680da6a7e2a875 (127.0.96.65:40485): Network error: Client connection negotiation failed: client connection to 127.0.96.65:40485: connect: Connection refused (error 111)
I20261120 03:03:36.513085  2086 leader_election.cc:304] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a157015bf640485fb805159465a9ab1b; no voters: 852e6c6e1ef54e5ebde6ab6f55b82b9c, 9ac1cc0b02fd4724b4680da6a7e2a875
W20261120 03:03:36.513708  2086 raft_consensus.cc:2902] T 178ee4361c534583b29d93a1e67d3c4f P a157015bf640485fb805159465a9ab1b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20261120 03:03:36.514537   385 tablet_replica.cc:333] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b: stopping tablet replica
I20261120 03:03:36.515241   385 raft_consensus.cc:2440] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.515856   385 raft_consensus.cc:2469] T ff883ba4b464463a9628eff440f1190a P a157015bf640485fb805159465a9ab1b [term 5 FOLLOWER]: Raft consensus is shut down!
W20261120 03:03:36.537401  2165 consensus_peers.cc:597] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c -> Peer a157015bf640485fb805159465a9ab1b (127.0.96.66:36367): Couldn't send request to peer a157015bf640485fb805159465a9ab1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.96.66:36367: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:36.546162   385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:03:36.559314   385 tablet_server.cc:179] TabletServer@127.0.96.67:0 shutting down...
W20261120 03:03:36.574368  2165 consensus_peers.cc:597] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c -> Peer a157015bf640485fb805159465a9ab1b (127.0.96.66:36367): Couldn't send request to peer a157015bf640485fb805159465a9ab1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.96.66:36367: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:36.585305   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:36.585899   385 tablet_replica.cc:333] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c: stopping tablet replica
I20261120 03:03:36.586660   385 raft_consensus.cc:2440] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 6 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.587358   385 raft_consensus.cc:2469] T 178ee4361c534583b29d93a1e67d3c4f P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 6 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.589263   385 tablet_replica.cc:333] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c: stopping tablet replica
I20261120 03:03:36.589651   385 raft_consensus.cc:2440] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 LEADER]: Raft consensus shutting down.
I20261120 03:03:36.590348   385 raft_consensus.cc:2469] T 791003cb42cf4e50a7666946f7ffad6b P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.592000   385 tablet_replica.cc:333] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c: stopping tablet replica
I20261120 03:03:36.592484   385 raft_consensus.cc:2440] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 LEADER]: Raft consensus shutting down.
I20261120 03:03:36.593086   385 raft_consensus.cc:2469] T ff883ba4b464463a9628eff440f1190a P 852e6c6e1ef54e5ebde6ab6f55b82b9c [term 5 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.614674   385 tablet_server.cc:196] TabletServer@127.0.96.67:0 shutdown complete.
I20261120 03:03:36.626992   385 master.cc:585] Master@127.0.96.126:43725 shutting down...
I20261120 03:03:36.647610   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 4 LEADER]: Raft consensus shutting down.
I20261120 03:03:36.648514   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.648893   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 764428dd82184375a0c80cbf46397e89: stopping tablet replica
I20261120 03:03:36.669073   385 master.cc:607] Master@127.0.96.126:43725 shutdown complete.
I20261120 03:03:36.685151   385 master.cc:585] Master@127.0.96.125:41159 shutting down...
I20261120 03:03:36.704452   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.704883   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P e68c557e0a6047f5956737cfd33fffb5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.705148   385 tablet_replica.cc:333] stopping tablet replica
I20261120 03:03:36.740841   385 master.cc:607] Master@127.0.96.125:41159 shutdown complete.
I20261120 03:03:36.753082   385 master.cc:585] Master@127.0.96.124:46415 shutting down...
I20261120 03:03:36.774307   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:36.774768   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 3a4598f24f6246008d66f45cff9355af [term 4 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:36.775199   385 tablet_replica.cc:333] stopping tablet replica
I20261120 03:03:36.793802   385 master.cc:607] Master@127.0.96.124:46415 shutdown complete.
I20261120 03:03:36.816525   385 test_util.cc:182] -----------------------------------------------
I20261120 03:03:36.816721   385 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1795143768723645-385-0
[  FAILED  ] TsTabletManagerITest.TestTableStats (35709 ms)
[ RUN      ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20261120 03:03:36.822628   385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:35193
I20261120 03:03:36.823696   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:36.829058  2674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:36.830194  2675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:36.833142  2677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:36.833853   385 server_base.cc:1061] running on GCE node
I20261120 03:03:36.835075   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:36.835322   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:36.835485   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143816835469 us; error 0 us; skew 500 ppm
I20261120 03:03:36.835983   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:36.838282   385 webserver.cc:577] Webserver started at http://127.0.96.126:44697/ using document root <none> and password file <none>
I20261120 03:03:36.838768   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:36.838964   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:36.839231   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:36.840350   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "48f97e77c6dc4c81886519ca702f6bc9"
format_stamp: "Formatted at 2026-11-20 03:03:36 on dist-test-slave-54pf"
I20261120 03:03:36.844494   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20261120 03:03:36.847424  2682 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:36.848256   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20261120 03:03:36.848469   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "48f97e77c6dc4c81886519ca702f6bc9"
format_stamp: "Formatted at 2026-11-20 03:03:36 on dist-test-slave-54pf"
I20261120 03:03:36.848675   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:36.873592   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:36.874595   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:36.908597   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:35193
I20261120 03:03:36.908674  2733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:35193 every 8 connection(s)
I20261120 03:03:36.912262  2734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:36.921344  2734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9: Bootstrap starting.
I20261120 03:03:36.925904  2734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:36.929539  2734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9: No bootstrap required, opened a new log
I20261120 03:03:36.931535  2734 raft_consensus.cc:366] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f97e77c6dc4c81886519ca702f6bc9" member_type: VOTER }
I20261120 03:03:36.931910  2734 raft_consensus.cc:392] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:36.932147  2734 raft_consensus.cc:787] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f97e77c6dc4c81886519ca702f6bc9, State: Initialized, Role: FOLLOWER
I20261120 03:03:36.932753  2734 consensus_queue.cc:261] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f97e77c6dc4c81886519ca702f6bc9" member_type: VOTER }
I20261120 03:03:36.933507  2734 raft_consensus.cc:407] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:36.933748  2734 raft_consensus.cc:534] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:36.933959  2734 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:36.934943  2734 raft_consensus.cc:556] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f97e77c6dc4c81886519ca702f6bc9" member_type: VOTER }
I20261120 03:03:36.938930  2734 leader_election.cc:304] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f97e77c6dc4c81886519ca702f6bc9; no voters: 
I20261120 03:03:36.939427  2734 leader_election.cc:290] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:03:36.939512  2737 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:36.940938  2737 raft_consensus.cc:744] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 48f97e77c6dc4c81886519ca702f6bc9, State: Running, Role: LEADER
I20261120 03:03:36.941531  2737 consensus_queue.cc:238] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f97e77c6dc4c81886519ca702f6bc9" member_type: VOTER }
I20261120 03:03:36.942318  2734 sys_catalog.cc:565] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:36.945531  2738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48f97e77c6dc4c81886519ca702f6bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f97e77c6dc4c81886519ca702f6bc9" member_type: VOTER } }
I20261120 03:03:36.946187  2738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:36.946025  2739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f97e77c6dc4c81886519ca702f6bc9. Latest consensus state: current_term: 1 leader_uuid: "48f97e77c6dc4c81886519ca702f6bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f97e77c6dc4c81886519ca702f6bc9" member_type: VOTER } }
I20261120 03:03:36.946525  2739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:36.953044  2744 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:36.957643  2744 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:36.961117   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:36.965432  2744 catalog_manager.cc:1398] Generated new cluster ID: b00adcad396f461a8dcbe4c4c06bf074
I20261120 03:03:36.965664  2744 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:36.989125  2744 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:36.990268  2744 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:37.002933  2744 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9: Generated new TSK 0
I20261120 03:03:37.003472  2744 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:37.026790   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:37.032438  2755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:37.033708  2756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:37.035487  2758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:37.035842   385 server_base.cc:1061] running on GCE node
I20261120 03:03:37.036856   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:37.037060   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:37.037237   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143817037220 us; error 0 us; skew 500 ppm
I20261120 03:03:37.037763   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:37.039880   385 webserver.cc:577] Webserver started at http://127.0.96.65:34409/ using document root <none> and password file <none>
I20261120 03:03:37.040369   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:37.040560   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:37.040823   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:37.041916   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "9f2bc43c379a4e998af2b0eee1b31ba6"
format_stamp: "Formatted at 2026-11-20 03:03:37 on dist-test-slave-54pf"
I20261120 03:03:37.046132   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20261120 03:03:37.049027  2763 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:37.049671   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20261120 03:03:37.050021   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "9f2bc43c379a4e998af2b0eee1b31ba6"
format_stamp: "Formatted at 2026-11-20 03:03:37 on dist-test-slave-54pf"
I20261120 03:03:37.050290   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:37.066728   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:37.067672   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:37.069028   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:37.070943   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:37.071110   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:37.071269   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:37.071444   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:37.107349   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:39839
I20261120 03:03:37.107445  2825 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:39839 every 8 connection(s)
I20261120 03:03:37.112056   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:37.118384  2830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:37.118839  2831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:37.122540  2833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:37.123454  2826 heartbeater.cc:345] Connected to a master server at 127.0.96.126:35193
I20261120 03:03:37.123790   385 server_base.cc:1061] running on GCE node
I20261120 03:03:37.123868  2826 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:37.124822  2826 heartbeater.cc:508] Master 127.0.96.126:35193 requested a full tablet report, sending...
I20261120 03:03:37.124840   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:37.125238   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:37.125404   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143817125373 us; error 0 us; skew 500 ppm
I20261120 03:03:37.126019   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:37.126928  2699 ts_manager.cc:210] Registered new tserver with Master: 9f2bc43c379a4e998af2b0eee1b31ba6 (127.0.96.65:39839)
I20261120 03:03:37.128551   385 webserver.cc:577] Webserver started at http://127.0.96.66:35119/ using document root <none> and password file <none>
I20261120 03:03:37.128690  2699 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52824
I20261120 03:03:37.129105   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:37.129338   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:37.129614   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:37.130702   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "b6200401445c426b8c6ffe3e81de3b57"
format_stamp: "Formatted at 2026-11-20 03:03:37 on dist-test-slave-54pf"
I20261120 03:03:37.134660   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20261120 03:03:37.137588  2838 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:37.138334   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20261120 03:03:37.138727   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "b6200401445c426b8c6ffe3e81de3b57"
format_stamp: "Formatted at 2026-11-20 03:03:37 on dist-test-slave-54pf"
I20261120 03:03:37.139051   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:37.152740   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:37.153805   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:37.155117   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:37.157248   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:37.157444   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:37.157697   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:37.157872   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:37.194317   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:38703
I20261120 03:03:37.194408  2900 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:38703 every 8 connection(s)
I20261120 03:03:37.198908   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:37.205538  2904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:37.206719  2905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:37.209313   385 server_base.cc:1061] running on GCE node
W20261120 03:03:37.209256  2907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:37.210557   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:37.210769   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:37.210985   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143817210966 us; error 0 us; skew 500 ppm
I20261120 03:03:37.211143  2901 heartbeater.cc:345] Connected to a master server at 127.0.96.126:35193
I20261120 03:03:37.211575  2901 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:37.211764   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:37.212502  2901 heartbeater.cc:508] Master 127.0.96.126:35193 requested a full tablet report, sending...
I20261120 03:03:37.214119   385 webserver.cc:577] Webserver started at http://127.0.96.67:43873/ using document root <none> and password file <none>
I20261120 03:03:37.214645   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:37.214692  2699 ts_manager.cc:210] Registered new tserver with Master: b6200401445c426b8c6ffe3e81de3b57 (127.0.96.66:38703)
I20261120 03:03:37.214955   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:37.215296   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:37.216234  2699 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52834
I20261120 03:03:37.216483   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-2-root/instance:
uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321"
format_stamp: "Formatted at 2026-11-20 03:03:37 on dist-test-slave-54pf"
I20261120 03:03:37.220948   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.004s
I20261120 03:03:37.223783  2912 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:37.224598   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20261120 03:03:37.224889   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-2-root
uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321"
format_stamp: "Formatted at 2026-11-20 03:03:37 on dist-test-slave-54pf"
I20261120 03:03:37.225142   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:37.243182   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:37.244294   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:37.245549   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:37.247562   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:37.247772   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:37.248001   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:37.248190   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20261120 03:03:37.285837   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.67:37007
I20261120 03:03:37.285914  2974 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.67:37007 every 8 connection(s)
I20261120 03:03:37.298624  2975 heartbeater.cc:345] Connected to a master server at 127.0.96.126:35193
I20261120 03:03:37.298906  2975 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:37.299487  2975 heartbeater.cc:508] Master 127.0.96.126:35193 requested a full tablet report, sending...
I20261120 03:03:37.301162  2699 ts_manager.cc:210] Registered new tserver with Master: 4e1e8c6f53ec4ad1bfd398c22a8a7321 (127.0.96.67:37007)
I20261120 03:03:37.302192   385 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01336271s
I20261120 03:03:37.302245  2699 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52848
I20261120 03:03:37.331578  2699 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52850:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
  }
}
W20261120 03:03:37.333089  2699 catalog_manager.cc:7244] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261120 03:03:37.379233  2866 tablet_service.cc:1495] Processing CreateTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 (DEFAULT_TABLE table=test-table [id=e97d03d945a34c338ddf9162958877e5]), partition=
I20261120 03:03:37.380316  2791 tablet_service.cc:1495] Processing CreateTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 (DEFAULT_TABLE table=test-table [id=e97d03d945a34c338ddf9162958877e5]), partition=
I20261120 03:03:37.380467  2866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62355b6698c54ab2b0ccd94c5749b743. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:37.381314  2791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62355b6698c54ab2b0ccd94c5749b743. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:37.381855  2940 tablet_service.cc:1495] Processing CreateTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 (DEFAULT_TABLE table=test-table [id=e97d03d945a34c338ddf9162958877e5]), partition=
I20261120 03:03:37.382730  2940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62355b6698c54ab2b0ccd94c5749b743. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:37.392630  2991 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Bootstrap starting.
I20261120 03:03:37.397866  2991 tablet_bootstrap.cc:654] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:37.401775  2991 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: No bootstrap required, opened a new log
I20261120 03:03:37.402084  2992 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: Bootstrap starting.
I20261120 03:03:37.402215  2991 ts_tablet_manager.cc:1449] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.009s
I20261120 03:03:37.402385  2993 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Bootstrap starting.
I20261120 03:03:37.404660  2991 raft_consensus.cc:366] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.405289  2991 raft_consensus.cc:392] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:37.405632  2991 raft_consensus.cc:787] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6200401445c426b8c6ffe3e81de3b57, State: Initialized, Role: FOLLOWER
I20261120 03:03:37.406520  2991 consensus_queue.cc:261] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.409225  2992 tablet_bootstrap.cc:654] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:37.409252  2993 tablet_bootstrap.cc:654] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:37.409535  2901 heartbeater.cc:500] Master 127.0.96.126:35193 was elected leader, sending a full tablet report...
I20261120 03:03:37.410061  2991 ts_tablet_manager.cc:1480] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
W20261120 03:03:37.410386  2991 ts_tablet_manager.cc:1503] Injecting 50ms of latency into OpenTablet
I20261120 03:03:37.414786  2992 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: No bootstrap required, opened a new log
I20261120 03:03:37.415238  2992 ts_tablet_manager.cc:1449] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: Time spent bootstrapping tablet: real 0.013s	user 0.004s	sys 0.009s
I20261120 03:03:37.415645  2993 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: No bootstrap required, opened a new log
I20261120 03:03:37.415990  2993 ts_tablet_manager.cc:1449] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Time spent bootstrapping tablet: real 0.014s	user 0.000s	sys 0.012s
I20261120 03:03:37.417281  2992 raft_consensus.cc:366] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.417899  2992 raft_consensus.cc:392] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:37.418247  2992 raft_consensus.cc:787] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2bc43c379a4e998af2b0eee1b31ba6, State: Initialized, Role: FOLLOWER
I20261120 03:03:37.418094  2993 raft_consensus.cc:366] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.418895  2993 raft_consensus.cc:392] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:37.419181  2993 raft_consensus.cc:787] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1e8c6f53ec4ad1bfd398c22a8a7321, State: Initialized, Role: FOLLOWER
I20261120 03:03:37.419127  2992 consensus_queue.cc:261] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.420181  2993 consensus_queue.cc:261] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.421643  2826 heartbeater.cc:500] Master 127.0.96.126:35193 was elected leader, sending a full tablet report...
I20261120 03:03:37.422281  2975 heartbeater.cc:500] Master 127.0.96.126:35193 was elected leader, sending a full tablet report...
I20261120 03:03:37.422595  2993 ts_tablet_manager.cc:1480] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
W20261120 03:03:37.422860  2993 ts_tablet_manager.cc:1503] Injecting 50ms of latency into OpenTablet
I20261120 03:03:37.423707  2992 ts_tablet_manager.cc:1480] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
W20261120 03:03:37.423967  2992 ts_tablet_manager.cc:1503] Injecting 50ms of latency into OpenTablet
W20261120 03:03:37.447980  2902 tablet.cc:2554] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261120 03:03:37.573923  2995 raft_consensus.cc:534] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:37.574303  2995 raft_consensus.cc:556] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.576249  2995 leader_election.cc:290] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e1e8c6f53ec4ad1bfd398c22a8a7321 (127.0.96.67:37007), 9f2bc43c379a4e998af2b0eee1b31ba6 (127.0.96.65:39839)
I20261120 03:03:37.583871  2950 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "62355b6698c54ab2b0ccd94c5749b743" candidate_uuid: "b6200401445c426b8c6ffe3e81de3b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" is_pre_election: true
I20261120 03:03:37.584713  2950 raft_consensus.cc:2062] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate b6200401445c426b8c6ffe3e81de3b57 in term 0
I20261120 03:03:37.584738  2999 raft_consensus.cc:534] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:37.585492  2999 raft_consensus.cc:556] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.585894  2801 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "62355b6698c54ab2b0ccd94c5749b743" candidate_uuid: "b6200401445c426b8c6ffe3e81de3b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" is_pre_election: true
I20261120 03:03:37.586064  2840 leader_election.cc:304] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e1e8c6f53ec4ad1bfd398c22a8a7321, b6200401445c426b8c6ffe3e81de3b57; no voters: 
I20261120 03:03:37.586648  2801 raft_consensus.cc:2062] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate b6200401445c426b8c6ffe3e81de3b57 in term 0
I20261120 03:03:37.586925  2995 raft_consensus.cc:3024] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:03:37.587292  2995 raft_consensus.cc:534] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:37.587633  2995 raft_consensus.cc:3296] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:37.588024  2999 leader_election.cc:290] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f2bc43c379a4e998af2b0eee1b31ba6 (127.0.96.65:39839), b6200401445c426b8c6ffe3e81de3b57 (127.0.96.66:38703)
I20261120 03:03:37.590991  2995 raft_consensus.cc:556] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.597860  2801 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "62355b6698c54ab2b0ccd94c5749b743" candidate_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" is_pre_election: true
I20261120 03:03:37.598455  2801 raft_consensus.cc:2062] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 4e1e8c6f53ec4ad1bfd398c22a8a7321 in term 0
I20261120 03:03:37.598994  2995 leader_election.cc:290] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [CANDIDATE]: Term 1 election: Requested vote from peers 4e1e8c6f53ec4ad1bfd398c22a8a7321 (127.0.96.67:37007), 9f2bc43c379a4e998af2b0eee1b31ba6 (127.0.96.65:39839)
I20261120 03:03:37.599531  2801 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "62355b6698c54ab2b0ccd94c5749b743" candidate_uuid: "b6200401445c426b8c6ffe3e81de3b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6"
I20261120 03:03:37.600111  2801 raft_consensus.cc:3296] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:37.599879  2876 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "62355b6698c54ab2b0ccd94c5749b743" candidate_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6200401445c426b8c6ffe3e81de3b57" is_pre_election: true
I20261120 03:03:37.600481  2950 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "62355b6698c54ab2b0ccd94c5749b743" candidate_uuid: "b6200401445c426b8c6ffe3e81de3b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321"
I20261120 03:03:37.601058  2950 raft_consensus.cc:3296] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:37.601157  2801 raft_consensus.cc:2062] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate b6200401445c426b8c6ffe3e81de3b57 in term 1
I20261120 03:03:37.601967  2950 raft_consensus.cc:2062] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate b6200401445c426b8c6ffe3e81de3b57 in term 1
I20261120 03:03:37.599869  2913 leader_election.cc:304] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e1e8c6f53ec4ad1bfd398c22a8a7321, 9f2bc43c379a4e998af2b0eee1b31ba6; no voters: 
I20261120 03:03:37.603581  3003 raft_consensus.cc:2984] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20261120 03:03:37.600992  2876 raft_consensus.cc:2628] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e1e8c6f53ec4ad1bfd398c22a8a7321 in current term 1: Already voted for candidate b6200401445c426b8c6ffe3e81de3b57 in this term.
I20261120 03:03:37.608165  2839 leader_election.cc:304] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f2bc43c379a4e998af2b0eee1b31ba6, b6200401445c426b8c6ffe3e81de3b57; no voters: 
I20261120 03:03:37.609030  2995 raft_consensus.cc:3024] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:37.609443  2995 raft_consensus.cc:744] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 1 LEADER]: Becoming Leader. State: Replica: b6200401445c426b8c6ffe3e81de3b57, State: Running, Role: LEADER
I20261120 03:03:37.610018  2995 consensus_queue.cc:238] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.617424  2995 consensus_queue.cc:1049] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:37.617172  2699 catalog_manager.cc:5856] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6200401445c426b8c6ffe3e81de3b57 (127.0.96.66). New cstate: current_term: 1 leader_uuid: "b6200401445c426b8c6ffe3e81de3b57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:37.618245  3002 consensus_queue.cc:1049] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:37.702065  2940 tablet_service.cc:1542] Processing DeleteTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:36696
I20261120 03:03:37.703416  3012 tablet_replica.cc:333] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: stopping tablet replica
I20261120 03:03:37.703969  3012 raft_consensus.cc:2440] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:37.704459  3012 raft_consensus.cc:2469] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:37.706770  3012 ts_tablet_manager.cc:1962] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20261120 03:03:37.716476  3012 ts_tablet_manager.cc:1975] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20261120 03:03:37.716735  3012 log.cc:1199] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Deleting WAL directory at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-2-root/wals/62355b6698c54ab2b0ccd94c5749b743
I20261120 03:03:37.720441  3013 ts_tablet_manager.cc:979] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Initiating tablet copy from peer b6200401445c426b8c6ffe3e81de3b57 (127.0.96.66:38703)
I20261120 03:03:37.722015  3013 tablet_copy_client.cc:323] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: tablet copy: Beginning tablet copy session from remote peer at address 127.0.96.66:38703
I20261120 03:03:37.723676  2886 tablet_copy_service.cc:140] P b6200401445c426b8c6ffe3e81de3b57: Received BeginTabletCopySession request for tablet 62355b6698c54ab2b0ccd94c5749b743 from peer 4e1e8c6f53ec4ad1bfd398c22a8a7321 ({username='slave'} at 127.0.0.1:46550)
I20261120 03:03:37.724087  2886 tablet_copy_service.cc:161] P b6200401445c426b8c6ffe3e81de3b57: Beginning new tablet copy session on tablet 62355b6698c54ab2b0ccd94c5749b743 from peer 4e1e8c6f53ec4ad1bfd398c22a8a7321 at {username='slave'} at 127.0.0.1:46550: session id = 4e1e8c6f53ec4ad1bfd398c22a8a7321-62355b6698c54ab2b0ccd94c5749b743
I20261120 03:03:37.728626  2886 tablet_copy_source_session.cc:215] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Tablet Copy: opened 0 blocks and 1 log segments
I20261120 03:03:37.730686  3013 ts_tablet_manager.cc:1962] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Deleting tablet data with delete state TABLET_DATA_COPYING
I20261120 03:03:37.739602  3013 ts_tablet_manager.cc:1975] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20261120 03:03:37.740206  3013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62355b6698c54ab2b0ccd94c5749b743. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:37.745079  3013 tablet_copy_client.cc:806] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: tablet copy: Starting download of 0 data blocks...
I20261120 03:03:37.745543  3013 tablet_copy_client.cc:670] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: tablet copy: Starting download of 1 WAL segments...
I20261120 03:03:37.746847  2699 catalog_manager.cc:2551] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52860:
table { table_id: "e97d03d945a34c338ddf9162958877e5" table_name: "test-table" }
I20261120 03:03:37.747483  2699 catalog_manager.cc:2799] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52860:
table { table_id: "e97d03d945a34c338ddf9162958877e5" table_name: "test-table" }
I20261120 03:03:37.748966  3013 tablet_copy_client.cc:538] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261120 03:03:37.754326  3013 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Bootstrap starting.
I20261120 03:03:37.757014  2699 catalog_manager.cc:6151] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9: Sending DeleteTablet for 3 replicas of tablet 62355b6698c54ab2b0ccd94c5749b743
I20261120 03:03:37.759042  2940 tablet_service.cc:1542] Processing DeleteTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-20 03:03:37 UTC) from {username='slave'} at 127.0.0.1:36680
I20261120 03:03:37.759325  2866 tablet_service.cc:1542] Processing DeleteTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-20 03:03:37 UTC) from {username='slave'} at 127.0.0.1:46534
I20261120 03:03:37.759738  2791 tablet_service.cc:1542] Processing DeleteTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-20 03:03:37 UTC) from {username='slave'} at 127.0.0.1:49972
W20261120 03:03:37.761077  2684 catalog_manager.cc:5061] TS 4e1e8c6f53ec4ad1bfd398c22a8a7321 (127.0.96.67:37007): delete failed for tablet 62355b6698c54ab2b0ccd94c5749b743 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62355b6698c54ab2b0ccd94c5749b743 already in progress: copying tablet
I20261120 03:03:37.763185  3015 tablet_replica.cc:333] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: stopping tablet replica
I20261120 03:03:37.763933  3016 tablet_replica.cc:333] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: stopping tablet replica
I20261120 03:03:37.763952  3015 raft_consensus.cc:2440] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:37.764940  3016 raft_consensus.cc:2440] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:37.765059  3015 raft_consensus.cc:2469] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:37.766001  3016 raft_consensus.cc:2469] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:37.768770  3016 ts_tablet_manager.cc:1962] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261120 03:03:37.769438  3015 ts_tablet_manager.cc:1962] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261120 03:03:37.776947  3013 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 03:03:37.777750  3013 tablet_bootstrap.cc:492] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Bootstrap complete.
I20261120 03:03:37.778242  3016 ts_tablet_manager.cc:1975] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261120 03:03:37.778398  3013 ts_tablet_manager.cc:1449] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Time spent bootstrapping tablet: real 0.024s	user 0.015s	sys 0.008s
I20261120 03:03:37.778694  3016 log.cc:1199] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: Deleting WAL directory at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-0-root/wals/62355b6698c54ab2b0ccd94c5749b743
I20261120 03:03:37.779461  3016 ts_tablet_manager.cc:1996] T 62355b6698c54ab2b0ccd94c5749b743 P 9f2bc43c379a4e998af2b0eee1b31ba6: Deleting consensus metadata
I20261120 03:03:37.781761  2683 catalog_manager.cc:5081] TS 9f2bc43c379a4e998af2b0eee1b31ba6 (127.0.96.65:39839): tablet 62355b6698c54ab2b0ccd94c5749b743 (table test-table [id=e97d03d945a34c338ddf9162958877e5]) successfully deleted
I20261120 03:03:37.781464  3013 raft_consensus.cc:366] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.782209  3013 raft_consensus.cc:787] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1e8c6f53ec4ad1bfd398c22a8a7321, State: Initialized, Role: FOLLOWER
I20261120 03:03:37.782871  3013 consensus_queue.cc:261] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1e8c6f53ec4ad1bfd398c22a8a7321" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 37007 } } peers { permanent_uuid: "9f2bc43c379a4e998af2b0eee1b31ba6" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 39839 } } peers { permanent_uuid: "b6200401445c426b8c6ffe3e81de3b57" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 38703 } }
I20261120 03:03:37.783362  3015 ts_tablet_manager.cc:1975] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261120 03:03:37.783725  3015 log.cc:1199] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Deleting WAL directory at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-1-root/wals/62355b6698c54ab2b0ccd94c5749b743
I20261120 03:03:37.784483  3015 ts_tablet_manager.cc:1996] T 62355b6698c54ab2b0ccd94c5749b743 P b6200401445c426b8c6ffe3e81de3b57: Deleting consensus metadata
I20261120 03:03:37.784538  3013 ts_tablet_manager.cc:1480] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
W20261120 03:03:37.784991  3013 ts_tablet_manager.cc:1503] Injecting 50ms of latency into OpenTablet
I20261120 03:03:37.786028  2685 catalog_manager.cc:5081] TS b6200401445c426b8c6ffe3e81de3b57 (127.0.96.66:38703): tablet 62355b6698c54ab2b0ccd94c5749b743 (table test-table [id=e97d03d945a34c338ddf9162958877e5]) successfully deleted
I20261120 03:03:37.788815  2940 tablet_service.cc:1542] Processing DeleteTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-20 03:03:37 UTC) from {username='slave'} at 127.0.0.1:36680
W20261120 03:03:37.790019  2684 catalog_manager.cc:5061] TS 4e1e8c6f53ec4ad1bfd398c22a8a7321 (127.0.96.67:37007): delete failed for tablet 62355b6698c54ab2b0ccd94c5749b743 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62355b6698c54ab2b0ccd94c5749b743 already in progress: copying tablet
I20261120 03:03:37.836783  2886 tablet_copy_service.cc:342] P b6200401445c426b8c6ffe3e81de3b57: Request end of tablet copy session 4e1e8c6f53ec4ad1bfd398c22a8a7321-62355b6698c54ab2b0ccd94c5749b743 received from {username='slave'} at 127.0.0.1:46550
I20261120 03:03:37.837286  2886 tablet_copy_service.cc:434] P b6200401445c426b8c6ffe3e81de3b57: ending tablet copy session 4e1e8c6f53ec4ad1bfd398c22a8a7321-62355b6698c54ab2b0ccd94c5749b743 on tablet 62355b6698c54ab2b0ccd94c5749b743 with peer 4e1e8c6f53ec4ad1bfd398c22a8a7321
I20261120 03:03:37.840396  2940 tablet_service.cc:1542] Processing DeleteTablet for tablet 62355b6698c54ab2b0ccd94c5749b743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-11-20 03:03:37 UTC) from {username='slave'} at 127.0.0.1:36680
I20261120 03:03:37.841112  3012 tablet_replica.cc:333] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: stopping tablet replica
I20261120 03:03:37.841753  3012 raft_consensus.cc:2440] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:37.842216  3012 raft_consensus.cc:2469] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:37.844332  3012 ts_tablet_manager.cc:1962] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261120 03:03:37.854117  3012 ts_tablet_manager.cc:1975] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261120 03:03:37.854394  3012 log.cc:1199] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Deleting WAL directory at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1795143768723645-385-0/minicluster-data/ts-2-root/wals/62355b6698c54ab2b0ccd94c5749b743
I20261120 03:03:37.854986  3012 ts_tablet_manager.cc:1996] T 62355b6698c54ab2b0ccd94c5749b743 P 4e1e8c6f53ec4ad1bfd398c22a8a7321: Deleting consensus metadata
I20261120 03:03:37.856973  2684 catalog_manager.cc:5081] TS 4e1e8c6f53ec4ad1bfd398c22a8a7321 (127.0.96.67:37007): tablet 62355b6698c54ab2b0ccd94c5749b743 (table test-table [id=e97d03d945a34c338ddf9162958877e5]) successfully deleted
I20261120 03:03:37.893677   385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:37.911146   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:37.927110   385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:37.935081   385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:03:37.949620   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:37.956969   385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:03:37.964517   385 tablet_server.cc:179] TabletServer@127.0.96.67:0 shutting down...
I20261120 03:03:37.979135   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:37.996016   385 tablet_server.cc:196] TabletServer@127.0.96.67:0 shutdown complete.
I20261120 03:03:38.004001   385 master.cc:585] Master@127.0.96.126:35193 shutting down...
I20261120 03:03:38.016253   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:38.016665   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:38.016921   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 48f97e77c6dc4c81886519ca702f6bc9: stopping tablet replica
I20261120 03:03:38.024362   385 master.cc:607] Master@127.0.96.126:35193 shutdown complete.
[       OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (1224 ms)
[----------] 4 tests from TsTabletManagerITest (43818 ms total)

[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN      ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20261120 03:03:38.046993   385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:33613
I20261120 03:03:38.047815   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:38.052239  3022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:38.053422  3023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:38.054831  3025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:38.055509   385 server_base.cc:1061] running on GCE node
I20261120 03:03:38.056279   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:38.056442   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:38.056550   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143818056539 us; error 0 us; skew 500 ppm
I20261120 03:03:38.056965   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:38.058912   385 webserver.cc:577] Webserver started at http://127.0.96.126:33819/ using document root <none> and password file <none>
I20261120 03:03:38.059326   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:38.059474   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:38.059664   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:38.060644   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d"
format_stamp: "Formatted at 2026-11-20 03:03:38 on dist-test-slave-54pf"
I20261120 03:03:38.064630   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20261120 03:03:38.067379  3030 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:38.068064   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20261120 03:03:38.068390   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d"
format_stamp: "Formatted at 2026-11-20 03:03:38 on dist-test-slave-54pf"
I20261120 03:03:38.068702   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:38.095054   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:38.095896   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:38.127470   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:33613
I20261120 03:03:38.127617  3081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:33613 every 8 connection(s)
I20261120 03:03:38.130923  3082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:38.140024  3082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d: Bootstrap starting.
I20261120 03:03:38.143985  3082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:38.147599  3082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d: No bootstrap required, opened a new log
I20261120 03:03:38.149374  3082 raft_consensus.cc:366] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" member_type: VOTER }
I20261120 03:03:38.149698  3082 raft_consensus.cc:392] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:38.149976  3082 raft_consensus.cc:787] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a9a0480eb54b3f8507798a5b4f1d7d, State: Initialized, Role: FOLLOWER
I20261120 03:03:38.150527  3082 consensus_queue.cc:261] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" member_type: VOTER }
I20261120 03:03:38.150995  3082 raft_consensus.cc:407] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:38.151206  3082 raft_consensus.cc:534] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:38.151487  3082 raft_consensus.cc:3296] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:38.152544  3082 raft_consensus.cc:556] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" member_type: VOTER }
I20261120 03:03:38.156459  3082 leader_election.cc:304] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0a9a0480eb54b3f8507798a5b4f1d7d; no voters: 
I20261120 03:03:38.156924  3082 leader_election.cc:290] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:03:38.157058  3085 raft_consensus.cc:3024] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:38.158406  3085 raft_consensus.cc:744] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 1 LEADER]: Becoming Leader. State: Replica: d0a9a0480eb54b3f8507798a5b4f1d7d, State: Running, Role: LEADER
I20261120 03:03:38.158990  3085 consensus_queue.cc:238] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" member_type: VOTER }
I20261120 03:03:38.159837  3082 sys_catalog.cc:565] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:38.161417  3086 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" member_type: VOTER } }
I20261120 03:03:38.161444  3087 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0a9a0480eb54b3f8507798a5b4f1d7d. Latest consensus state: current_term: 1 leader_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a9a0480eb54b3f8507798a5b4f1d7d" member_type: VOTER } }
I20261120 03:03:38.162071  3086 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:38.162211  3087 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:38.164747  3089 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:38.169283  3089 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:38.174937   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:38.177464  3089 catalog_manager.cc:1398] Generated new cluster ID: 7d429e6644994dc8b3a50487f73512e6
I20261120 03:03:38.177699  3089 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:38.194279  3089 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:38.195394  3089 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:38.207247  3089 catalog_manager.cc:6237] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d: Generated new TSK 0
I20261120 03:03:38.207782  3089 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:38.241671   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:38.246934  3103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:38.247908  3104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:38.249339  3106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:38.249595   385 server_base.cc:1061] running on GCE node
I20261120 03:03:38.250379   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:38.250617   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:38.250833   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143818250813 us; error 0 us; skew 500 ppm
I20261120 03:03:38.251739   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:38.253851   385 webserver.cc:577] Webserver started at http://127.0.96.65:44629/ using document root <none> and password file <none>
I20261120 03:03:38.254345   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:38.254554   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:38.254818   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:38.255822   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "2b4b6609ec75466faa98fdf15bc21513"
format_stamp: "Formatted at 2026-11-20 03:03:38 on dist-test-slave-54pf"
I20261120 03:03:38.259929   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20261120 03:03:38.262929  3111 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:38.263665   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20261120 03:03:38.263950   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "2b4b6609ec75466faa98fdf15bc21513"
format_stamp: "Formatted at 2026-11-20 03:03:38 on dist-test-slave-54pf"
I20261120 03:03:38.264287   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:38.279279   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:38.280294   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:38.284619   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:38.284965   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.001s
I20261120 03:03:38.285368   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:38.285679   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:38.333323   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:37323
I20261120 03:03:38.333424  3181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:37323 every 8 connection(s)
I20261120 03:03:38.337821  3182 heartbeater.cc:345] Connected to a master server at 127.0.96.126:33613
I20261120 03:03:38.338173  3182 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:38.338856  3182 heartbeater.cc:508] Master 127.0.96.126:33613 requested a full tablet report, sending...
I20261120 03:03:38.340669  3047 ts_manager.cc:210] Registered new tserver with Master: 2b4b6609ec75466faa98fdf15bc21513 (127.0.96.65:37323)
I20261120 03:03:38.341233   385 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004807337s
I20261120 03:03:38.342628  3047 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34144
I20261120 03:03:38.369163  3144 tablet_service.cc:1495] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261120 03:03:38.370262  3144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:38.380915  3192 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Bootstrap starting.
I20261120 03:03:38.385248  3192 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:38.388710  3192 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: No bootstrap required, opened a new log
I20261120 03:03:38.389050  3192 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20261120 03:03:38.390914  3192 raft_consensus.cc:366] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.391351  3192 raft_consensus.cc:392] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:38.391592  3192 raft_consensus.cc:787] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4b6609ec75466faa98fdf15bc21513, State: Initialized, Role: FOLLOWER
I20261120 03:03:38.392146  3192 consensus_queue.cc:261] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.392652  3192 raft_consensus.cc:407] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:38.392879  3192 raft_consensus.cc:534] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:38.393142  3192 raft_consensus.cc:3296] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:38.394240  3192 raft_consensus.cc:556] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.398432  3192 leader_election.cc:304] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b4b6609ec75466faa98fdf15bc21513; no voters: 
I20261120 03:03:38.398914  3192 leader_election.cc:290] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:03:38.399055  3194 raft_consensus.cc:3024] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:38.400314  3194 raft_consensus.cc:744] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 LEADER]: Becoming Leader. State: Replica: 2b4b6609ec75466faa98fdf15bc21513, State: Running, Role: LEADER
I20261120 03:03:38.400795  3182 heartbeater.cc:500] Master 127.0.96.126:33613 was elected leader, sending a full tablet report...
I20261120 03:03:38.401002  3194 consensus_queue.cc:238] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.401332  3192 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20261120 03:03:38.404860  3196 tablet_replica.cc:442] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: TxnStatusTablet state changed. Reason: New leader 2b4b6609ec75466faa98fdf15bc21513. Latest consensus state: current_term: 1 leader_uuid: "2b4b6609ec75466faa98fdf15bc21513" committed_config { opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } } }
I20261120 03:03:38.405422  3196 tablet_replica.cc:445] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: This TxnStatusTablet replica's current role is: LEADER
I20261120 03:03:38.406159  3195 tablet_replica.cc:442] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b4b6609ec75466faa98fdf15bc21513" committed_config { opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } } }
I20261120 03:03:38.406800  3195 tablet_replica.cc:445] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: This TxnStatusTablet replica's current role is: LEADER
W20261120 03:03:38.408792  3047 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:38.411069  3199 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261120 03:03:38.411535  3199 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20261120 03:03:38.413298  3047 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:38.444834   385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:38.467916   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:38.468705   385 tablet_replica.cc:333] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: stopping tablet replica
I20261120 03:03:38.469285   385 raft_consensus.cc:2440] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:38.469785   385 raft_consensus.cc:2469] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:38.491005   385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:38.501700   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:38.507189  3202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:38.507602  3203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:38.510063   385 server_base.cc:1061] running on GCE node
W20261120 03:03:38.510900  3206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:38.511818   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:38.512076   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:38.512297   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143818512279 us; error 0 us; skew 500 ppm
I20261120 03:03:38.513002   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:38.515527   385 webserver.cc:577] Webserver started at http://127.0.96.65:44629/ using document root <none> and password file <none>
I20261120 03:03:38.516067   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:38.516287   385 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261120 03:03:38.520737   385 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.001s
I20261120 03:03:38.523923  3210 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:38.525939   385 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20261120 03:03:38.526365   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "2b4b6609ec75466faa98fdf15bc21513"
format_stamp: "Formatted at 2026-11-20 03:03:38 on dist-test-slave-54pf"
I20261120 03:03:38.526681   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:38.538625   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:38.539753   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:38.547371  3219 ts_tablet_manager.cc:594] Loading tablet metadata (0/1 complete)
I20261120 03:03:38.554382   385 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261120 03:03:38.554675   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.008s	user 0.001s	sys 0.001s
I20261120 03:03:38.555027   385 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261120 03:03:38.561659   385 ts_tablet_manager.cc:662] Registered 1 tablets
I20261120 03:03:38.561928   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.007s	user 0.007s	sys 0.000s
I20261120 03:03:38.562053  3219 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Bootstrap starting.
I20261120 03:03:38.603374  3219 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 03:03:38.604401  3219 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Bootstrap complete.
I20261120 03:03:38.605239  3219 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Time spent bootstrapping tablet: real 0.043s	user 0.030s	sys 0.012s
I20261120 03:03:38.608026  3219 raft_consensus.cc:366] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.608671  3219 raft_consensus.cc:787] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4b6609ec75466faa98fdf15bc21513, State: Initialized, Role: FOLLOWER
I20261120 03:03:38.609582  3219 consensus_queue.cc:261] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.610260  3219 raft_consensus.cc:407] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:38.610569  3219 raft_consensus.cc:534] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:38.610997  3219 raft_consensus.cc:3296] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:38.623193  3219 raft_consensus.cc:556] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.623845  3219 leader_election.cc:304] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b4b6609ec75466faa98fdf15bc21513; no voters: 
I20261120 03:03:38.624598  3219 leader_election.cc:290] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [CANDIDATE]: Term 2 election: Requested vote from peers 
I20261120 03:03:38.625162  3251 raft_consensus.cc:3024] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:38.631089  3219 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513: Time spent starting tablet: real 0.025s	user 0.011s	sys 0.007s
I20261120 03:03:38.633127  3251 raft_consensus.cc:744] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 2 LEADER]: Becoming Leader. State: Replica: 2b4b6609ec75466faa98fdf15bc21513, State: Running, Role: LEADER
I20261120 03:03:38.633842  3251 consensus_queue.cc:238] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } }
I20261120 03:03:38.636710  3264 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b4b6609ec75466faa98fdf15bc21513" committed_config { opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } } }
I20261120 03:03:38.636857  3265 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 2b4b6609ec75466faa98fdf15bc21513. Latest consensus state: current_term: 2 leader_uuid: "2b4b6609ec75466faa98fdf15bc21513" committed_config { opid_index: -1 peers { permanent_uuid: "2b4b6609ec75466faa98fdf15bc21513" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37323 } } }
I20261120 03:03:38.637264  3265 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20261120 03:03:38.637217  3264 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20261120 03:03:38.639274  3269 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261120 03:03:38.639799  3269 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20261120 03:03:38.665566   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:37323
I20261120 03:03:38.665700  3287 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:37323 every 8 connection(s)
I20261120 03:03:38.671871  3288 heartbeater.cc:345] Connected to a master server at 127.0.96.126:33613
I20261120 03:03:38.672273  3288 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:38.672922  3288 heartbeater.cc:508] Master 127.0.96.126:33613 requested a full tablet report, sending...
I20261120 03:03:38.673813   385 tablet_server.cc:179] TabletServer@127.0.96.65:37323 shutting down...
I20261120 03:03:38.676519  3047 ts_manager.cc:210] Re-registered known tserver with Master: 2b4b6609ec75466faa98fdf15bc21513 (127.0.96.65:37323)
W20261120 03:03:38.677172  3047 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:38.679832  3047 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34174
I20261120 03:03:38.682637  3288 heartbeater.cc:500] Master 127.0.96.126:33613 was elected leader, sending a full tablet report...
W20261120 03:03:38.685861  3047 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:38.697319   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:38.698107   385 tablet_replica.cc:333] stopping tablet replica
I20261120 03:03:38.698545   385 raft_consensus.cc:2440] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 2 LEADER]: Raft consensus shutting down.
I20261120 03:03:38.698938   385 raft_consensus.cc:2469] T 11111111111111111111111111111111 P 2b4b6609ec75466faa98fdf15bc21513 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:38.709548   385 tablet_server.cc:196] TabletServer@127.0.96.65:37323 shutdown complete.
I20261120 03:03:38.719607   385 master.cc:585] Master@127.0.96.126:33613 shutting down...
I20261120 03:03:38.736292   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:38.736702   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:38.737025   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P d0a9a0480eb54b3f8507798a5b4f1d7d: stopping tablet replica
I20261120 03:03:38.754426   385 master.cc:607] Master@127.0.96.126:33613 shutdown complete.
[       OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (728 ms)
[ RUN      ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20261120 03:03:38.777001   385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:44779
I20261120 03:03:38.777968   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:38.782974  3294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:38.784624  3295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:38.785967  3297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:38.786686   385 server_base.cc:1061] running on GCE node
I20261120 03:03:38.787498   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:38.787734   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:38.787912   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143818787898 us; error 0 us; skew 500 ppm
I20261120 03:03:38.788625   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:38.791148   385 webserver.cc:577] Webserver started at http://127.0.96.126:34183/ using document root <none> and password file <none>
I20261120 03:03:38.791749   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:38.791965   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:38.792290   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:38.793642   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "90084b6d6a1149049fe264596414f97f"
format_stamp: "Formatted at 2026-11-20 03:03:38 on dist-test-slave-54pf"
I20261120 03:03:38.799250   385 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20261120 03:03:38.803155  3302 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:38.804008   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20261120 03:03:38.804379   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "90084b6d6a1149049fe264596414f97f"
format_stamp: "Formatted at 2026-11-20 03:03:38 on dist-test-slave-54pf"
I20261120 03:03:38.804715   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:38.839732   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:38.841032   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:38.883744   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:44779
I20261120 03:03:38.883816  3353 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:44779 every 8 connection(s)
I20261120 03:03:38.888427  3354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:38.897773  3354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f: Bootstrap starting.
I20261120 03:03:38.901922  3354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:38.905615  3354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f: No bootstrap required, opened a new log
I20261120 03:03:38.907755  3354 raft_consensus.cc:366] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90084b6d6a1149049fe264596414f97f" member_type: VOTER }
I20261120 03:03:38.908149  3354 raft_consensus.cc:392] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:38.908399  3354 raft_consensus.cc:787] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90084b6d6a1149049fe264596414f97f, State: Initialized, Role: FOLLOWER
I20261120 03:03:38.908974  3354 consensus_queue.cc:261] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90084b6d6a1149049fe264596414f97f" member_type: VOTER }
I20261120 03:03:38.909403  3354 raft_consensus.cc:407] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:38.909636  3354 raft_consensus.cc:534] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:38.909888  3354 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:38.911149  3354 raft_consensus.cc:556] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90084b6d6a1149049fe264596414f97f" member_type: VOTER }
I20261120 03:03:38.915311  3354 leader_election.cc:304] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90084b6d6a1149049fe264596414f97f; no voters: 
I20261120 03:03:38.915831  3354 leader_election.cc:290] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:03:38.915951  3357 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:38.917809  3357 raft_consensus.cc:744] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 1 LEADER]: Becoming Leader. State: Replica: 90084b6d6a1149049fe264596414f97f, State: Running, Role: LEADER
I20261120 03:03:38.918542  3357 consensus_queue.cc:238] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90084b6d6a1149049fe264596414f97f" member_type: VOTER }
I20261120 03:03:38.919188  3354 sys_catalog.cc:565] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:38.921428  3358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90084b6d6a1149049fe264596414f97f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90084b6d6a1149049fe264596414f97f" member_type: VOTER } }
I20261120 03:03:38.921506  3359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90084b6d6a1149049fe264596414f97f. Latest consensus state: current_term: 1 leader_uuid: "90084b6d6a1149049fe264596414f97f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90084b6d6a1149049fe264596414f97f" member_type: VOTER } }
I20261120 03:03:38.921998  3358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:38.922163  3359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:38.923926  3362 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:38.932618  3362 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:38.947317   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:38.948166  3362 catalog_manager.cc:1398] Generated new cluster ID: 3d21ad82e2d44867b991835b46a85e85
I20261120 03:03:38.948463  3362 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:38.977007  3362 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:38.978188  3362 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:38.993410  3362 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f: Generated new TSK 0
I20261120 03:03:38.993928  3362 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:39.013108   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:39.019004  3375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:39.020277  3376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:39.022747  3378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:39.023042   385 server_base.cc:1061] running on GCE node
I20261120 03:03:39.023939   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:39.024176   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:39.024366   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143819024347 us; error 0 us; skew 500 ppm
I20261120 03:03:39.024910   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:39.027498   385 webserver.cc:577] Webserver started at http://127.0.96.65:33121/ using document root <none> and password file <none>
I20261120 03:03:39.027962   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:39.028180   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:39.028451   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:39.029439   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "29f19b523fc04e318d7afa0b6a545ab0"
format_stamp: "Formatted at 2026-11-20 03:03:39 on dist-test-slave-54pf"
I20261120 03:03:39.034052   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.001s
I20261120 03:03:39.037467  3383 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:39.038319   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20261120 03:03:39.038600   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "29f19b523fc04e318d7afa0b6a545ab0"
format_stamp: "Formatted at 2026-11-20 03:03:39 on dist-test-slave-54pf"
I20261120 03:03:39.038880   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:39.056347   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:39.057201   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:39.069103   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:39.069528   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:39.069921   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:39.070230   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:39.134486   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:43325
I20261120 03:03:39.134596  3453 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:43325 every 8 connection(s)
I20261120 03:03:39.138975   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261120 03:03:39.139176  3454 heartbeater.cc:345] Connected to a master server at 127.0.96.126:44779
I20261120 03:03:39.139545  3454 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:39.140400  3454 heartbeater.cc:508] Master 127.0.96.126:44779 requested a full tablet report, sending...
I20261120 03:03:39.142834  3319 ts_manager.cc:210] Registered new tserver with Master: 29f19b523fc04e318d7afa0b6a545ab0 (127.0.96.65:43325)
I20261120 03:03:39.145284  3319 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51690
W20261120 03:03:39.145722  3456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:39.148631  3457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:39.151209   385 server_base.cc:1061] running on GCE node
W20261120 03:03:39.151683  3459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:39.152599   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:39.152792   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:39.152961   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143819152945 us; error 0 us; skew 500 ppm
I20261120 03:03:39.153401   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:39.155987   385 webserver.cc:577] Webserver started at http://127.0.96.66:43279/ using document root <none> and password file <none>
I20261120 03:03:39.156498   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:39.156669   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:39.156922   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:39.157923   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "2bc2e534f6364cfc80a46267fed30937"
format_stamp: "Formatted at 2026-11-20 03:03:39 on dist-test-slave-54pf"
I20261120 03:03:39.162351   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20261120 03:03:39.165693  3464 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:39.166433   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20261120 03:03:39.166700   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "2bc2e534f6364cfc80a46267fed30937"
format_stamp: "Formatted at 2026-11-20 03:03:39 on dist-test-slave-54pf"
I20261120 03:03:39.166980   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:39.177129   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:39.178004   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:39.183730   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:39.184079   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20261120 03:03:39.184404   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:39.184581   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:39.301927   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:42795
I20261120 03:03:39.302165  3534 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:42795 every 8 connection(s)
I20261120 03:03:39.309206  3535 heartbeater.cc:345] Connected to a master server at 127.0.96.126:44779
I20261120 03:03:39.309597  3535 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:39.310452  3535 heartbeater.cc:508] Master 127.0.96.126:44779 requested a full tablet report, sending...
I20261120 03:03:39.312848  3319 ts_manager.cc:210] Registered new tserver with Master: 2bc2e534f6364cfc80a46267fed30937 (127.0.96.66:42795)
I20261120 03:03:39.313560   385 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.006529321s
I20261120 03:03:39.314637  3319 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51696
I20261120 03:03:39.352620  3419 tablet_service.cc:1495] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261120 03:03:39.354125  3419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:39.366775  3545 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Bootstrap starting.
I20261120 03:03:39.371899  3545 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:39.376322  3545 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: No bootstrap required, opened a new log
I20261120 03:03:39.376689  3545 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.001s
I20261120 03:03:39.378970  3545 raft_consensus.cc:366] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.379684  3545 raft_consensus.cc:392] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:39.379925  3545 raft_consensus.cc:787] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f19b523fc04e318d7afa0b6a545ab0, State: Initialized, Role: FOLLOWER
I20261120 03:03:39.380532  3545 consensus_queue.cc:261] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.381021  3545 raft_consensus.cc:407] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:39.381251  3545 raft_consensus.cc:534] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:39.381569  3545 raft_consensus.cc:3296] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:39.382936  3545 raft_consensus.cc:556] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.389446  3545 leader_election.cc:304] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f19b523fc04e318d7afa0b6a545ab0; no voters: 
I20261120 03:03:39.389961  3545 leader_election.cc:290] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:03:39.390188  3547 raft_consensus.cc:3024] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:39.392256  3454 heartbeater.cc:500] Master 127.0.96.126:44779 was elected leader, sending a full tablet report...
I20261120 03:03:39.392671  3545 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.004s
I20261120 03:03:39.393440  3547 raft_consensus.cc:744] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 29f19b523fc04e318d7afa0b6a545ab0, State: Running, Role: LEADER
I20261120 03:03:39.394208  3547 consensus_queue.cc:238] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.399405  3549 tablet_replica.cc:442] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: TxnStatusTablet state changed. Reason: New leader 29f19b523fc04e318d7afa0b6a545ab0. Latest consensus state: current_term: 1 leader_uuid: "29f19b523fc04e318d7afa0b6a545ab0" committed_config { opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } } }
I20261120 03:03:39.400182  3549 tablet_replica.cc:445] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: This TxnStatusTablet replica's current role is: LEADER
W20261120 03:03:39.401614  3319 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:39.404492  3548 tablet_replica.cc:442] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29f19b523fc04e318d7afa0b6a545ab0" committed_config { opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } } }
I20261120 03:03:39.405079  3548 tablet_replica.cc:445] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: This TxnStatusTablet replica's current role is: LEADER
W20261120 03:03:39.413125  3319 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:39.413573  3552 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261120 03:03:39.414068  3552 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20261120 03:03:39.466661  3556 ts_tablet_manager.cc:979] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: Initiating tablet copy from peer 29f19b523fc04e318d7afa0b6a545ab0 (127.0.96.65:43325)
I20261120 03:03:39.468468  3556 tablet_copy_client.cc:323] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: tablet copy: Beginning tablet copy session from remote peer at address 127.0.96.65:43325
I20261120 03:03:39.478209  3439 tablet_copy_service.cc:140] P 29f19b523fc04e318d7afa0b6a545ab0: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer 2bc2e534f6364cfc80a46267fed30937 ({username='slave'} at 127.0.0.1:33944)
I20261120 03:03:39.478766  3439 tablet_copy_service.cc:161] P 29f19b523fc04e318d7afa0b6a545ab0: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer 2bc2e534f6364cfc80a46267fed30937 at {username='slave'} at 127.0.0.1:33944: session id = 2bc2e534f6364cfc80a46267fed30937-11111111111111111111111111111111
I20261120 03:03:39.486583  3439 tablet_copy_source_session.cc:215] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Tablet Copy: opened 0 blocks and 1 log segments
I20261120 03:03:39.489581  3556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:39.501308  3556 tablet_copy_client.cc:806] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: tablet copy: Starting download of 0 data blocks...
I20261120 03:03:39.501874  3556 tablet_copy_client.cc:670] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: tablet copy: Starting download of 1 WAL segments...
I20261120 03:03:39.505625  3556 tablet_copy_client.cc:538] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261120 03:03:39.511534  3556 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: Bootstrap starting.
I20261120 03:03:39.535190  3556 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 03:03:39.535859  3556 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: Bootstrap complete.
I20261120 03:03:39.536470  3556 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: Time spent bootstrapping tablet: real 0.025s	user 0.023s	sys 0.005s
I20261120 03:03:39.538587  3556 raft_consensus.cc:366] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.538949  3556 raft_consensus.cc:787] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 2bc2e534f6364cfc80a46267fed30937, State: Initialized, Role: NON_PARTICIPANT
I20261120 03:03:39.539464  3556 consensus_queue.cc:261] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.541438  3535 heartbeater.cc:500] Master 127.0.96.126:44779 was elected leader, sending a full tablet report...
I20261120 03:03:39.541457  3559 tablet_replica.cc:442] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } } }
I20261120 03:03:39.541993  3556 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20261120 03:03:39.542106  3559 tablet_replica.cc:445] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20261120 03:03:39.543777  3439 tablet_copy_service.cc:342] P 29f19b523fc04e318d7afa0b6a545ab0: Request end of tablet copy session 2bc2e534f6364cfc80a46267fed30937-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:33944
I20261120 03:03:39.544296  3439 tablet_copy_service.cc:434] P 29f19b523fc04e318d7afa0b6a545ab0: ending tablet copy session 2bc2e534f6364cfc80a46267fed30937-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer 2bc2e534f6364cfc80a46267fed30937
W20261120 03:03:39.544634  3319 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
W20261120 03:03:39.548262  3319 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:39.566248   385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:39.591985   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:39.592936   385 tablet_replica.cc:333] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: stopping tablet replica
I20261120 03:03:39.593420   385 raft_consensus.cc:2440] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:39.593847   385 raft_consensus.cc:2469] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:39.614725   385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:39.627701   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:39.633776  3560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:39.635381  3561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:39.637706  3563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:39.637914   385 server_base.cc:1061] running on GCE node
I20261120 03:03:39.638983   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:39.639195   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:39.639429   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143819639413 us; error 0 us; skew 500 ppm
I20261120 03:03:39.639978   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:39.642980   385 webserver.cc:577] Webserver started at http://127.0.96.65:33121/ using document root <none> and password file <none>
I20261120 03:03:39.643568   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:39.643781   385 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261120 03:03:39.648080   385 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.001s
I20261120 03:03:39.651216  3568 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:39.652027   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20261120 03:03:39.652382   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "29f19b523fc04e318d7afa0b6a545ab0"
format_stamp: "Formatted at 2026-11-20 03:03:39 on dist-test-slave-54pf"
I20261120 03:03:39.652789   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:39.699798   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:39.701328   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:39.709100  3577 ts_tablet_manager.cc:594] Loading tablet metadata (0/1 complete)
I20261120 03:03:39.716747   385 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261120 03:03:39.717092   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.009s	user 0.000s	sys 0.002s
I20261120 03:03:39.717418   385 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261120 03:03:39.723649   385 ts_tablet_manager.cc:662] Registered 1 tablets
I20261120 03:03:39.723934   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.007s	user 0.004s	sys 0.002s
I20261120 03:03:39.723986  3577 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Bootstrap starting.
I20261120 03:03:39.774514  3577 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 03:03:39.775728  3577 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Bootstrap complete.
I20261120 03:03:39.776535  3577 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Time spent bootstrapping tablet: real 0.053s	user 0.038s	sys 0.011s
I20261120 03:03:39.780279  3577 raft_consensus.cc:366] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.780925  3577 raft_consensus.cc:787] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f19b523fc04e318d7afa0b6a545ab0, State: Initialized, Role: FOLLOWER
I20261120 03:03:39.781551  3577 consensus_queue.cc:261] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.782045  3577 raft_consensus.cc:407] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:39.782302  3577 raft_consensus.cc:534] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:39.782603  3577 raft_consensus.cc:3296] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:39.783998  3577 raft_consensus.cc:556] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.788662  3577 leader_election.cc:304] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f19b523fc04e318d7afa0b6a545ab0; no voters: 
I20261120 03:03:39.789165  3577 leader_election.cc:290] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [CANDIDATE]: Term 2 election: Requested vote from peers 
I20261120 03:03:39.789247  3603 raft_consensus.cc:3024] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:39.798045  3577 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0: Time spent starting tablet: real 0.021s	user 0.011s	sys 0.003s
I20261120 03:03:39.799409  3603 raft_consensus.cc:744] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 2 LEADER]: Becoming Leader. State: Replica: 29f19b523fc04e318d7afa0b6a545ab0, State: Running, Role: LEADER
I20261120 03:03:39.800287  3603 consensus_queue.cc:238] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } }
I20261120 03:03:39.803408  3607 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 29f19b523fc04e318d7afa0b6a545ab0. Latest consensus state: current_term: 2 leader_uuid: "29f19b523fc04e318d7afa0b6a545ab0" committed_config { opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } } }
I20261120 03:03:39.803500  3606 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "29f19b523fc04e318d7afa0b6a545ab0" committed_config { opid_index: -1 peers { permanent_uuid: "29f19b523fc04e318d7afa0b6a545ab0" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43325 } } }
I20261120 03:03:39.803918  3607 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20261120 03:03:39.805014  3606 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20261120 03:03:39.807024  3612 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261120 03:03:39.807510  3612 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20261120 03:03:39.836958   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:43325
I20261120 03:03:39.837103  3645 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:43325 every 8 connection(s)
I20261120 03:03:39.842549  3646 heartbeater.cc:345] Connected to a master server at 127.0.96.126:44779
I20261120 03:03:39.843006  3646 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:39.843951  3646 heartbeater.cc:508] Master 127.0.96.126:44779 requested a full tablet report, sending...
I20261120 03:03:39.845338   385 tablet_server.cc:179] TabletServer@127.0.96.65:43325 shutting down...
I20261120 03:03:39.847255  3319 ts_manager.cc:210] Re-registered known tserver with Master: 29f19b523fc04e318d7afa0b6a545ab0 (127.0.96.65:43325)
W20261120 03:03:39.847904  3319 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:39.849491  3319 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51724
I20261120 03:03:39.853502  3646 heartbeater.cc:500] Master 127.0.96.126:44779 was elected leader, sending a full tablet report...
W20261120 03:03:39.860013  3319 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:39.868870   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:39.869601   385 tablet_replica.cc:333] stopping tablet replica
I20261120 03:03:39.870154   385 raft_consensus.cc:2440] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 2 LEADER]: Raft consensus shutting down.
I20261120 03:03:39.870601   385 raft_consensus.cc:2469] T 11111111111111111111111111111111 P 29f19b523fc04e318d7afa0b6a545ab0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:39.882125   385 tablet_server.cc:196] TabletServer@127.0.96.65:43325 shutdown complete.
I20261120 03:03:39.894556   385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:03:39.913226   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:39.914215   385 tablet_replica.cc:333] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937: stopping tablet replica
I20261120 03:03:39.914847   385 raft_consensus.cc:2440] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20261120 03:03:39.915282   385 raft_consensus.cc:2469] T 11111111111111111111111111111111 P 2bc2e534f6364cfc80a46267fed30937 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20261120 03:03:39.934993   385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:03:39.946018   385 master.cc:585] Master@127.0.96.126:44779 shutting down...
I20261120 03:03:39.968099   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:39.968645   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:39.969000   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 90084b6d6a1149049fe264596414f97f: stopping tablet replica
I20261120 03:03:39.987951   385 master.cc:607] Master@127.0.96.126:44779 shutdown complete.
[       OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (1244 ms)
[ RUN      ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20261120 03:03:40.021370   385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:35375
I20261120 03:03:40.022516   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:40.027840  3652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.029366  3653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.030719  3655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:40.031698   385 server_base.cc:1061] running on GCE node
I20261120 03:03:40.032490   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:40.032692   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:40.032866   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143820032850 us; error 0 us; skew 500 ppm
I20261120 03:03:40.033371   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:40.035655   385 webserver.cc:577] Webserver started at http://127.0.96.126:39129/ using document root <none> and password file <none>
I20261120 03:03:40.036157   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:40.036360   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:40.036638   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:40.037850   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "a20fe6cb0199458c8e038b7230fbd4d2"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.042358   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20261120 03:03:40.045689  3660 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:40.046444   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20261120 03:03:40.046707   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "a20fe6cb0199458c8e038b7230fbd4d2"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.047015   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:40.070524   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:40.072292   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:40.120599   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:35375
I20261120 03:03:40.120684  3711 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:35375 every 8 connection(s)
I20261120 03:03:40.124970  3712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:40.139037  3712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2: Bootstrap starting.
I20261120 03:03:40.145249  3712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:40.150049  3712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2: No bootstrap required, opened a new log
I20261120 03:03:40.152752  3712 raft_consensus.cc:366] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" member_type: VOTER }
I20261120 03:03:40.153303  3712 raft_consensus.cc:392] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:40.153590  3712 raft_consensus.cc:787] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20fe6cb0199458c8e038b7230fbd4d2, State: Initialized, Role: FOLLOWER
I20261120 03:03:40.154253  3712 consensus_queue.cc:261] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" member_type: VOTER }
I20261120 03:03:40.154816  3712 raft_consensus.cc:407] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:40.155122  3712 raft_consensus.cc:534] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:40.155423  3712 raft_consensus.cc:3296] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:40.156715  3712 raft_consensus.cc:556] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" member_type: VOTER }
I20261120 03:03:40.160808  3712 leader_election.cc:304] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a20fe6cb0199458c8e038b7230fbd4d2; no voters: 
I20261120 03:03:40.161425  3712 leader_election.cc:290] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:03:40.161507  3715 raft_consensus.cc:3024] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:40.162766  3715 raft_consensus.cc:744] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 1 LEADER]: Becoming Leader. State: Replica: a20fe6cb0199458c8e038b7230fbd4d2, State: Running, Role: LEADER
I20261120 03:03:40.163530  3715 consensus_queue.cc:238] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" member_type: VOTER }
I20261120 03:03:40.165468  3712 sys_catalog.cc:565] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:40.166340  3717 sys_catalog.cc:455] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a20fe6cb0199458c8e038b7230fbd4d2. Latest consensus state: current_term: 1 leader_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" member_type: VOTER } }
I20261120 03:03:40.166312  3716 sys_catalog.cc:455] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20fe6cb0199458c8e038b7230fbd4d2" member_type: VOTER } }
I20261120 03:03:40.166976  3716 sys_catalog.cc:458] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:40.166953  3717 sys_catalog.cc:458] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:40.168705  3719 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:40.177263  3719 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:40.189292  3719 catalog_manager.cc:1398] Generated new cluster ID: a71f8f1cb0f0458c886cfa8b99499885
I20261120 03:03:40.189536  3719 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:40.195376   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:40.212721  3719 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:40.213783  3719 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:40.227721  3719 catalog_manager.cc:6237] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2: Generated new TSK 0
I20261120 03:03:40.228312  3719 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:40.261360   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:40.267423  3733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.269688  3734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.272063  3736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:40.272440   385 server_base.cc:1061] running on GCE node
I20261120 03:03:40.273270   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:40.273483   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:40.273661   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143820273644 us; error 0 us; skew 500 ppm
I20261120 03:03:40.274350   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:40.277109   385 webserver.cc:577] Webserver started at http://127.0.96.65:40217/ using document root <none> and password file <none>
I20261120 03:03:40.277555   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:40.277722   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:40.277981   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:40.278906   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "dd8b334dc99d45409536cffa87977dad"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.283298   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20261120 03:03:40.286818  3741 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:40.287652   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20261120 03:03:40.287907   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "dd8b334dc99d45409536cffa87977dad"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.288173   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:40.305012   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:40.305881   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:40.313352   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:40.313606   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:40.313922   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:40.314111   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:40.391618   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:43037
I20261120 03:03:40.391712  3811 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:43037 every 8 connection(s)
I20261120 03:03:40.396592  3812 heartbeater.cc:345] Connected to a master server at 127.0.96.126:35375
I20261120 03:03:40.396984  3812 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:40.397778  3812 heartbeater.cc:508] Master 127.0.96.126:35375 requested a full tablet report, sending...
I20261120 03:03:40.399880  3677 ts_manager.cc:210] Registered new tserver with Master: dd8b334dc99d45409536cffa87977dad (127.0.96.65:43037)
I20261120 03:03:40.400933   385 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.005905478s
I20261120 03:03:40.402272  3677 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33544
I20261120 03:03:40.433107  3774 tablet_service.cc:1495] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261120 03:03:40.434216  3774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:40.444790  3822 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: Bootstrap starting.
I20261120 03:03:40.449271  3822 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:40.453147  3822 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: No bootstrap required, opened a new log
I20261120 03:03:40.453522  3822 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.003s
I20261120 03:03:40.455349  3822 raft_consensus.cc:366] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "dd8b334dc99d45409536cffa87977dad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43037 } }
I20261120 03:03:40.455751  3822 raft_consensus.cc:392] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:40.455981  3822 raft_consensus.cc:787] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8b334dc99d45409536cffa87977dad, State: Initialized, Role: FOLLOWER
I20261120 03:03:40.456599  3822 consensus_queue.cc:261] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "dd8b334dc99d45409536cffa87977dad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43037 } }
I20261120 03:03:40.457053  3822 raft_consensus.cc:407] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:40.457298  3822 raft_consensus.cc:534] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:40.457556  3822 raft_consensus.cc:3296] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:40.458889  3822 raft_consensus.cc:556] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "dd8b334dc99d45409536cffa87977dad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43037 } }
I20261120 03:03:40.463342  3822 leader_election.cc:304] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd8b334dc99d45409536cffa87977dad; no voters: 
I20261120 03:03:40.463927  3822 leader_election.cc:290] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:03:40.464041  3824 raft_consensus.cc:3024] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:40.465787  3812 heartbeater.cc:500] Master 127.0.96.126:35375 was elected leader, sending a full tablet report...
I20261120 03:03:40.466665  3822 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20261120 03:03:40.467075  3824 raft_consensus.cc:744] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 1 LEADER]: Becoming Leader. State: Replica: dd8b334dc99d45409536cffa87977dad, State: Running, Role: LEADER
I20261120 03:03:40.467767  3824 consensus_queue.cc:238] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "dd8b334dc99d45409536cffa87977dad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43037 } }
I20261120 03:03:40.471000  3826 tablet_replica.cc:442] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: TxnStatusTablet state changed. Reason: New leader dd8b334dc99d45409536cffa87977dad. Latest consensus state: current_term: 1 leader_uuid: "dd8b334dc99d45409536cffa87977dad" committed_config { opid_index: -1 peers { permanent_uuid: "dd8b334dc99d45409536cffa87977dad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43037 } } }
I20261120 03:03:40.471580  3826 tablet_replica.cc:445] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: This TxnStatusTablet replica's current role is: LEADER
W20261120 03:03:40.473856  3677 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:40.477933  3825 tablet_replica.cc:442] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd8b334dc99d45409536cffa87977dad" committed_config { opid_index: -1 peers { permanent_uuid: "dd8b334dc99d45409536cffa87977dad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 43037 } } }
I20261120 03:03:40.478482  3825 tablet_replica.cc:445] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: This TxnStatusTablet replica's current role is: LEADER
I20261120 03:03:40.480010  3829 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261120 03:03:40.480456  3829 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20261120 03:03:40.482400  3677 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
W20261120 03:03:40.527911  3783 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20261120 03:03:40.528252  3783 txn_status_manager.cc:244] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20261120 03:03:40.528581  3783 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20261120 03:03:40.538007  3783 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20261120 03:03:40.572774  3783 txn_status_manager.cc:244] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20261120 03:03:40.573120  3783 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20261120 03:03:40.580839  3783 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20261120 03:03:40.590636  3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:43762
W20261120 03:03:40.592785  3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:43762
W20261120 03:03:40.594630  3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:43762
W20261120 03:03:40.596457  3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:43762
W20261120 03:03:40.598248  3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:43762
W20261120 03:03:40.600021  3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:43762
W20261120 03:03:40.609162  3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:43774
W20261120 03:03:40.610951  3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:43774
W20261120 03:03:40.612811  3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:43774
W20261120 03:03:40.614547  3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:43774
W20261120 03:03:40.616415  3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:43774
W20261120 03:03:40.618290  3774 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:43774
I20261120 03:03:40.621930   385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:40.644902   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:40.645676   385 tablet_replica.cc:333] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad: stopping tablet replica
I20261120 03:03:40.646243   385 raft_consensus.cc:2440] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:40.646761   385 raft_consensus.cc:2469] T 11111111111111111111111111111111 P dd8b334dc99d45409536cffa87977dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:40.660984   385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:40.672039   385 master.cc:585] Master@127.0.96.126:35375 shutting down...
I20261120 03:03:40.692554   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:40.693075   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:40.693516   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P a20fe6cb0199458c8e038b7230fbd4d2: stopping tablet replica
I20261120 03:03:40.702705   385 master.cc:607] Master@127.0.96.126:35375 shutdown complete.
[       OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (711 ms)
[ RUN      ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20261120 03:03:40.734160   385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:41445
I20261120 03:03:40.735435   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:40.740741  3837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.741715  3838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.744571  3840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:40.745334   385 server_base.cc:1061] running on GCE node
I20261120 03:03:40.746071   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:40.746281   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:40.746438   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143820746424 us; error 0 us; skew 500 ppm
I20261120 03:03:40.747010   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:40.749281   385 webserver.cc:577] Webserver started at http://127.0.96.126:42135/ using document root <none> and password file <none>
I20261120 03:03:40.749832   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:40.750028   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:40.750314   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:40.751595   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "ff5af720f1ad4794b00b97e9e46d90da"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.757112   385 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20261120 03:03:40.760917  3845 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:40.761646   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.004s	sys 0.000s
I20261120 03:03:40.761932   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "ff5af720f1ad4794b00b97e9e46d90da"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.762225   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:40.791435   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:40.792690   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:40.835601   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:41445
I20261120 03:03:40.835657  3896 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:41445 every 8 connection(s)
I20261120 03:03:40.840356  3897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:40.850013  3897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da: Bootstrap starting.
I20261120 03:03:40.854591  3897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:40.858253  3897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da: No bootstrap required, opened a new log
I20261120 03:03:40.860522  3897 raft_consensus.cc:366] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5af720f1ad4794b00b97e9e46d90da" member_type: VOTER }
I20261120 03:03:40.860880  3897 raft_consensus.cc:392] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:40.861083  3897 raft_consensus.cc:787] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5af720f1ad4794b00b97e9e46d90da, State: Initialized, Role: FOLLOWER
I20261120 03:03:40.861697  3897 consensus_queue.cc:261] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5af720f1ad4794b00b97e9e46d90da" member_type: VOTER }
I20261120 03:03:40.862141  3897 raft_consensus.cc:407] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:40.862380  3897 raft_consensus.cc:534] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:40.862653  3897 raft_consensus.cc:3296] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:40.863829  3897 raft_consensus.cc:556] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5af720f1ad4794b00b97e9e46d90da" member_type: VOTER }
I20261120 03:03:40.869557  3897 leader_election.cc:304] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff5af720f1ad4794b00b97e9e46d90da; no voters: 
I20261120 03:03:40.870023  3897 leader_election.cc:290] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:03:40.870152  3900 raft_consensus.cc:3024] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:40.873136  3900 raft_consensus.cc:744] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 1 LEADER]: Becoming Leader. State: Replica: ff5af720f1ad4794b00b97e9e46d90da, State: Running, Role: LEADER
I20261120 03:03:40.873337  3897 sys_catalog.cc:565] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:40.873766  3900 consensus_queue.cc:238] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5af720f1ad4794b00b97e9e46d90da" member_type: VOTER }
I20261120 03:03:40.878285  3901 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff5af720f1ad4794b00b97e9e46d90da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5af720f1ad4794b00b97e9e46d90da" member_type: VOTER } }
I20261120 03:03:40.879467  3902 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff5af720f1ad4794b00b97e9e46d90da. Latest consensus state: current_term: 1 leader_uuid: "ff5af720f1ad4794b00b97e9e46d90da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5af720f1ad4794b00b97e9e46d90da" member_type: VOTER } }
I20261120 03:03:40.880240  3902 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:40.880100  3901 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:40.889783  3906 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:40.896148  3906 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:40.898020   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:40.905746  3906 catalog_manager.cc:1398] Generated new cluster ID: c3d34109b22143c69ca30bfd3b28dbfd
I20261120 03:03:40.906001  3906 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:40.925205  3906 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:40.926249  3906 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:40.937126  3906 catalog_manager.cc:6237] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da: Generated new TSK 0
I20261120 03:03:40.937680  3906 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:40.964715   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:40.970655  3918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.971565  3919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:40.974201  3921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:40.974577   385 server_base.cc:1061] running on GCE node
I20261120 03:03:40.975626   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:40.975873   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:40.976063   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143820976046 us; error 0 us; skew 500 ppm
I20261120 03:03:40.976656   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:40.978881   385 webserver.cc:577] Webserver started at http://127.0.96.65:33811/ using document root <none> and password file <none>
I20261120 03:03:40.979331   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:40.979532   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:40.979795   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:40.980865   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "2a5e1457634e4ac09477344f614be8ad"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.985024   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20261120 03:03:40.987910  3926 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:40.988605   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20261120 03:03:40.988885   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "2a5e1457634e4ac09477344f614be8ad"
format_stamp: "Formatted at 2026-11-20 03:03:40 on dist-test-slave-54pf"
I20261120 03:03:40.989163   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:40.998661   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:40.999577   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:41.004293   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:41.004714   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:41.005105   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:41.005369   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:41.049824   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:44759
I20261120 03:03:41.049921  3996 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:44759 every 8 connection(s)
I20261120 03:03:41.054438  3997 heartbeater.cc:345] Connected to a master server at 127.0.96.126:41445
I20261120 03:03:41.054793  3997 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:41.055516  3997 heartbeater.cc:508] Master 127.0.96.126:41445 requested a full tablet report, sending...
I20261120 03:03:41.057473  3862 ts_manager.cc:210] Registered new tserver with Master: 2a5e1457634e4ac09477344f614be8ad (127.0.96.65:44759)
I20261120 03:03:41.057881   385 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004832045s
I20261120 03:03:41.059098  3862 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55114
I20261120 03:03:41.086515  3962 tablet_service.cc:1495] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261120 03:03:41.087561  3962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:41.097600  4007 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: Bootstrap starting.
I20261120 03:03:41.101473  4007 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:41.104816  4007 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: No bootstrap required, opened a new log
I20261120 03:03:41.105176  4007 ts_tablet_manager.cc:1449] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20261120 03:03:41.106896  4007 raft_consensus.cc:366] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.107267  4007 raft_consensus.cc:392] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:41.107491  4007 raft_consensus.cc:787] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5e1457634e4ac09477344f614be8ad, State: Initialized, Role: FOLLOWER
I20261120 03:03:41.107956  4007 consensus_queue.cc:261] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.108381  4007 raft_consensus.cc:407] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:41.108585  4007 raft_consensus.cc:534] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:41.108831  4007 raft_consensus.cc:3296] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:41.109830  4007 raft_consensus.cc:556] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.114051  4007 leader_election.cc:304] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a5e1457634e4ac09477344f614be8ad; no voters: 
I20261120 03:03:41.114545  4007 leader_election.cc:290] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:03:41.114624  4009 raft_consensus.cc:3024] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:41.115950  4009 raft_consensus.cc:744] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 1 LEADER]: Becoming Leader. State: Replica: 2a5e1457634e4ac09477344f614be8ad, State: Running, Role: LEADER
I20261120 03:03:41.116426  3997 heartbeater.cc:500] Master 127.0.96.126:41445 was elected leader, sending a full tablet report...
I20261120 03:03:41.116847  4007 ts_tablet_manager.cc:1480] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20261120 03:03:41.116638  4009 consensus_queue.cc:238] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.119297  4010 tablet_replica.cc:442] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a5e1457634e4ac09477344f614be8ad" committed_config { opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } } }
I20261120 03:03:41.119836  4010 tablet_replica.cc:445] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: This TxnStatusTablet replica's current role is: LEADER
I20261120 03:03:41.119666  4011 tablet_replica.cc:442] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: TxnStatusTablet state changed. Reason: New leader 2a5e1457634e4ac09477344f614be8ad. Latest consensus state: current_term: 1 leader_uuid: "2a5e1457634e4ac09477344f614be8ad" committed_config { opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } } }
I20261120 03:03:41.120205  4011 tablet_replica.cc:445] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: This TxnStatusTablet replica's current role is: LEADER
W20261120 03:03:41.122058  3862 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:41.122148  4013 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20261120 03:03:41.122555  4013 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20261120 03:03:41.125686  3862 catalog_manager.cc:5689] Ignoring report from unknown tablet 11111111111111111111111111111111
I20261120 03:03:41.142879  3962 tablet_service.cc:1495] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20261120 03:03:41.143806  3962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:41.153038  4007 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad: Bootstrap starting.
I20261120 03:03:41.157117  4007 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:41.160583  4007 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad: No bootstrap required, opened a new log
I20261120 03:03:41.160971  4007 ts_tablet_manager.cc:1449] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20261120 03:03:41.162745  4007 raft_consensus.cc:366] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.163146  4007 raft_consensus.cc:392] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:41.163371  4007 raft_consensus.cc:787] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5e1457634e4ac09477344f614be8ad, State: Initialized, Role: FOLLOWER
I20261120 03:03:41.163842  4007 consensus_queue.cc:261] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.164400  4007 raft_consensus.cc:407] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:41.164644  4007 raft_consensus.cc:534] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:41.164911  4007 raft_consensus.cc:3296] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:41.165422  4007 raft_consensus.cc:556] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.169600  4007 leader_election.cc:304] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a5e1457634e4ac09477344f614be8ad; no voters: 
I20261120 03:03:41.170068  4007 leader_election.cc:290] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:03:41.170171  4009 raft_consensus.cc:3024] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:41.170687  4009 raft_consensus.cc:744] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 1 LEADER]: Becoming Leader. State: Replica: 2a5e1457634e4ac09477344f614be8ad, State: Running, Role: LEADER
I20261120 03:03:41.171363  4009 consensus_queue.cc:238] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2a5e1457634e4ac09477344f614be8ad" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 44759 } }
I20261120 03:03:41.171980  4007 ts_tablet_manager.cc:1480] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
W20261120 03:03:41.176295  3862 catalog_manager.cc:5689] Ignoring report from unknown tablet 22222222222222222222222222222222
W20261120 03:03:41.179756  3862 catalog_manager.cc:5689] Ignoring report from unknown tablet 22222222222222222222222222222222
I20261120 03:03:41.200241   385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:41.218407   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:41.219049   385 tablet_replica.cc:333] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad: stopping tablet replica
I20261120 03:03:41.219542   385 raft_consensus.cc:2440] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:41.219938   385 raft_consensus.cc:2469] T 22222222222222222222222222222222 P 2a5e1457634e4ac09477344f614be8ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:41.222157   385 tablet_replica.cc:333] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad: stopping tablet replica
I20261120 03:03:41.222592   385 raft_consensus.cc:2440] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:41.222961   385 raft_consensus.cc:2469] T 11111111111111111111111111111111 P 2a5e1457634e4ac09477344f614be8ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:41.233364   385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:41.243028   385 master.cc:585] Master@127.0.96.126:41445 shutting down...
I20261120 03:03:41.256150   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:41.256629   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:41.257108   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P ff5af720f1ad4794b00b97e9e46d90da: stopping tablet replica
I20261120 03:03:41.275498   385 master.cc:607] Master@127.0.96.126:41445 shutdown complete.
[       OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (565 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (3250 ms total)

[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN      ] FailedTabletsAreReplacedITest.OneReplica/0
I20261120 03:03:41.298225   385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:33659
I20261120 03:03:41.299171   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:41.303613  4023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:41.304847  4024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:41.306308   385 server_base.cc:1061] running on GCE node
W20261120 03:03:41.307052  4026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:41.307755   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:41.307962   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:41.308156   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143821308099 us; error 0 us; skew 500 ppm
I20261120 03:03:41.308710   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:41.310799   385 webserver.cc:577] Webserver started at http://127.0.96.126:33651/ using document root <none> and password file <none>
I20261120 03:03:41.311371   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:41.311558   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:41.311825   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:41.312868   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "7d9d0a04125446849a404f868bc1cf29"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.317116   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20261120 03:03:41.320200  4031 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:41.320843   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20261120 03:03:41.321163   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "7d9d0a04125446849a404f868bc1cf29"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.321416   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:41.332790   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:41.333626   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:41.365540   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:33659
I20261120 03:03:41.365624  4082 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:33659 every 8 connection(s)
I20261120 03:03:41.369102  4083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:41.378253  4083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29: Bootstrap starting.
I20261120 03:03:41.382076  4083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:41.385432  4083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29: No bootstrap required, opened a new log
I20261120 03:03:41.387146  4083 raft_consensus.cc:366] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9d0a04125446849a404f868bc1cf29" member_type: VOTER }
I20261120 03:03:41.387478  4083 raft_consensus.cc:392] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:41.387645  4083 raft_consensus.cc:787] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9d0a04125446849a404f868bc1cf29, State: Initialized, Role: FOLLOWER
I20261120 03:03:41.388090  4083 consensus_queue.cc:261] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9d0a04125446849a404f868bc1cf29" member_type: VOTER }
I20261120 03:03:41.388540  4083 raft_consensus.cc:407] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:41.388722  4083 raft_consensus.cc:534] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:41.388914  4083 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:41.389823  4083 raft_consensus.cc:556] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9d0a04125446849a404f868bc1cf29" member_type: VOTER }
I20261120 03:03:41.394058  4083 leader_election.cc:304] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d9d0a04125446849a404f868bc1cf29; no voters: 
I20261120 03:03:41.394572  4083 leader_election.cc:290] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:03:41.394675  4086 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:41.396109  4086 raft_consensus.cc:744] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 1 LEADER]: Becoming Leader. State: Replica: 7d9d0a04125446849a404f868bc1cf29, State: Running, Role: LEADER
I20261120 03:03:41.396765  4086 consensus_queue.cc:238] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9d0a04125446849a404f868bc1cf29" member_type: VOTER }
I20261120 03:03:41.397466  4083 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:41.398994  4087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d9d0a04125446849a404f868bc1cf29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9d0a04125446849a404f868bc1cf29" member_type: VOTER } }
I20261120 03:03:41.400556  4087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:41.399967  4088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d9d0a04125446849a404f868bc1cf29. Latest consensus state: current_term: 1 leader_uuid: "7d9d0a04125446849a404f868bc1cf29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9d0a04125446849a404f868bc1cf29" member_type: VOTER } }
I20261120 03:03:41.401556  4088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:41.404686  4093 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:41.408788  4093 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:41.411461   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:41.416814  4093 catalog_manager.cc:1398] Generated new cluster ID: cd3a48158fbc443ca1058733e20dfd72
I20261120 03:03:41.417060  4093 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:41.437521  4093 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:41.438637  4093 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:41.452047  4093 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29: Generated new TSK 0
I20261120 03:03:41.452585  4093 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:41.477620   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:41.483075  4104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:41.484180  4105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:41.486274  4107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:41.486425   385 server_base.cc:1061] running on GCE node
I20261120 03:03:41.487218   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:41.487444   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:41.487633   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143821487599 us; error 0 us; skew 500 ppm
I20261120 03:03:41.488238   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:41.490120   385 webserver.cc:577] Webserver started at http://127.0.96.65:35397/ using document root <none> and password file <none>
I20261120 03:03:41.490566   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:41.490819   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:41.491082   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:41.492120   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "5686819a367f4567b741ac52b51c69a3"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.496287   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20261120 03:03:41.499235  4112 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:41.499929   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20261120 03:03:41.500211   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "5686819a367f4567b741ac52b51c69a3"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.500460   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:41.528260   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:41.529086   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:41.530413   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:41.532538   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:41.532917   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:41.533177   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:41.533346   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:41.567986   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:37193
I20261120 03:03:41.568096  4174 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:37193 every 8 connection(s)
I20261120 03:03:41.572611   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:41.578867  4179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:41.579675  4180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:41.582686  4182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:41.583205   385 server_base.cc:1061] running on GCE node
I20261120 03:03:41.583747  4175 heartbeater.cc:345] Connected to a master server at 127.0.96.126:33659
I20261120 03:03:41.584174  4175 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:41.584396   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:41.584645   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:41.584848   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143821584830 us; error 0 us; skew 500 ppm
I20261120 03:03:41.584931  4175 heartbeater.cc:508] Master 127.0.96.126:33659 requested a full tablet report, sending...
I20261120 03:03:41.585547   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:41.587034  4048 ts_manager.cc:210] Registered new tserver with Master: 5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193)
I20261120 03:03:41.588285   385 webserver.cc:577] Webserver started at http://127.0.96.66:42431/ using document root <none> and password file <none>
I20261120 03:03:41.588747   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:41.588922   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:41.588912  4048 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44338
I20261120 03:03:41.589334   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:41.590415   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "e554bd281f3844c098fb147463c08a84"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.594368   385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.001s
I20261120 03:03:41.597323  4187 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:41.598014   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20261120 03:03:41.598266   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "e554bd281f3844c098fb147463c08a84"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.598501   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:41.606246   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:41.607110   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:41.608439   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:41.610357   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:41.610548   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:41.610761   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:41.610935   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:41.644881   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:45765
I20261120 03:03:41.644966  4249 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:45765 every 8 connection(s)
I20261120 03:03:41.649176   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:41.656430  4253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:41.656517  4254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:41.659888  4250 heartbeater.cc:345] Connected to a master server at 127.0.96.126:33659
I20261120 03:03:41.660374  4250 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:41.660789   385 server_base.cc:1061] running on GCE node
W20261120 03:03:41.660785  4256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:41.661147  4250 heartbeater.cc:508] Master 127.0.96.126:33659 requested a full tablet report, sending...
I20261120 03:03:41.661780   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:41.662009   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:41.662176   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143821662163 us; error 0 us; skew 500 ppm
I20261120 03:03:41.662904   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:41.663110  4048 ts_manager.cc:210] Registered new tserver with Master: e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765)
I20261120 03:03:41.664584  4048 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44344
I20261120 03:03:41.665344   385 webserver.cc:577] Webserver started at http://127.0.96.67:41609/ using document root <none> and password file <none>
I20261120 03:03:41.665824   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:41.666026   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:41.666280   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:41.667299   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-2-root/instance:
uuid: "c94544a8d7604d15b3a451f3657b06a7"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.671358   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20261120 03:03:41.674188  4261 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:41.674993   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20261120 03:03:41.675215   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-2-root
uuid: "c94544a8d7604d15b3a451f3657b06a7"
format_stamp: "Formatted at 2026-11-20 03:03:41 on dist-test-slave-54pf"
I20261120 03:03:41.675431   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:41.685007   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:41.685797   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:41.686866   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:41.688683   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:41.688843   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:41.689095   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:41.689227   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:41.723927   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.67:45369
I20261120 03:03:41.724020  4323 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.67:45369 every 8 connection(s)
I20261120 03:03:41.736411  4324 heartbeater.cc:345] Connected to a master server at 127.0.96.126:33659
I20261120 03:03:41.736753  4324 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:41.737376  4324 heartbeater.cc:508] Master 127.0.96.126:33659 requested a full tablet report, sending...
I20261120 03:03:41.738946  4048 ts_manager.cc:210] Registered new tserver with Master: c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369)
I20261120 03:03:41.739969   385 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013123899s
I20261120 03:03:41.740010  4048 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44350
I20261120 03:03:41.776002  4048 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44368:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20261120 03:03:41.819103  4289 tablet_service.cc:1495] Processing CreateTablet for tablet 961865cf1efc4c71a4a12606b68f48d5 (DEFAULT_TABLE table=test-workload [id=74fbc5a4629d4a4aa0d813b8d2120ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:41.820379  4289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961865cf1efc4c71a4a12606b68f48d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:41.821167  4140 tablet_service.cc:1495] Processing CreateTablet for tablet 961865cf1efc4c71a4a12606b68f48d5 (DEFAULT_TABLE table=test-workload [id=74fbc5a4629d4a4aa0d813b8d2120ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:41.822368  4140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961865cf1efc4c71a4a12606b68f48d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:41.830118  4215 tablet_service.cc:1495] Processing CreateTablet for tablet 961865cf1efc4c71a4a12606b68f48d5 (DEFAULT_TABLE table=test-workload [id=74fbc5a4629d4a4aa0d813b8d2120ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:41.831264  4215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961865cf1efc4c71a4a12606b68f48d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:41.836215  4344 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Bootstrap starting.
I20261120 03:03:41.841174  4345 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3: Bootstrap starting.
I20261120 03:03:41.843219  4344 tablet_bootstrap.cc:654] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:41.847738  4345 tablet_bootstrap.cc:654] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:41.848511  4346 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: Bootstrap starting.
I20261120 03:03:41.851770  4344 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: No bootstrap required, opened a new log
I20261120 03:03:41.852245  4344 ts_tablet_manager.cc:1449] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Time spent bootstrapping tablet: real 0.016s	user 0.009s	sys 0.004s
I20261120 03:03:41.852553  4345 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3: No bootstrap required, opened a new log
I20261120 03:03:41.852994  4345 ts_tablet_manager.cc:1449] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.009s
I20261120 03:03:41.854224  4346 tablet_bootstrap.cc:654] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:41.855042  4344 raft_consensus.cc:366] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.855865  4344 raft_consensus.cc:392] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:41.855720  4345 raft_consensus.cc:366] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.856241  4344 raft_consensus.cc:787] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94544a8d7604d15b3a451f3657b06a7, State: Initialized, Role: FOLLOWER
I20261120 03:03:41.856462  4345 raft_consensus.cc:392] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:41.856789  4345 raft_consensus.cc:787] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5686819a367f4567b741ac52b51c69a3, State: Initialized, Role: FOLLOWER
I20261120 03:03:41.857089  4344 consensus_queue.cc:261] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.857481  4345 consensus_queue.cc:261] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.860028  4175 heartbeater.cc:500] Master 127.0.96.126:33659 was elected leader, sending a full tablet report...
I20261120 03:03:41.860716  4345 ts_tablet_manager.cc:1480] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20261120 03:03:41.862874  4324 heartbeater.cc:500] Master 127.0.96.126:33659 was elected leader, sending a full tablet report...
I20261120 03:03:41.863874  4344 ts_tablet_manager.cc:1480] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.008s
I20261120 03:03:41.864183  4346 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: No bootstrap required, opened a new log
I20261120 03:03:41.864670  4346 ts_tablet_manager.cc:1449] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.008s
I20261120 03:03:41.867216  4346 raft_consensus.cc:366] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.867726  4346 raft_consensus.cc:392] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:41.867964  4346 raft_consensus.cc:787] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e554bd281f3844c098fb147463c08a84, State: Initialized, Role: FOLLOWER
I20261120 03:03:41.868701  4346 consensus_queue.cc:261] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.871793  4346 ts_tablet_manager.cc:1480] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.002s
I20261120 03:03:41.872277  4250 heartbeater.cc:500] Master 127.0.96.126:33659 was elected leader, sending a full tablet report...
I20261120 03:03:41.923336  4350 raft_consensus.cc:534] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:41.923703  4350 raft_consensus.cc:556] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.925637  4350 leader_election.cc:290] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193), e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765)
I20261120 03:03:41.934147  4150 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "c94544a8d7604d15b3a451f3657b06a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5686819a367f4567b741ac52b51c69a3" is_pre_election: true
I20261120 03:03:41.934981  4150 raft_consensus.cc:2062] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate c94544a8d7604d15b3a451f3657b06a7 in term 0
I20261120 03:03:41.935662  4225 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "c94544a8d7604d15b3a451f3657b06a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e554bd281f3844c098fb147463c08a84" is_pre_election: true
I20261120 03:03:41.935972  4264 leader_election.cc:304] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5686819a367f4567b741ac52b51c69a3, c94544a8d7604d15b3a451f3657b06a7; no voters: 
I20261120 03:03:41.936336  4225 raft_consensus.cc:2062] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate c94544a8d7604d15b3a451f3657b06a7 in term 0
I20261120 03:03:41.936715  4350 raft_consensus.cc:3024] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:03:41.937031  4350 raft_consensus.cc:534] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:41.937307  4350 raft_consensus.cc:3296] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:41.938187  4350 raft_consensus.cc:556] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.942989  4350 leader_election.cc:290] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [CANDIDATE]: Term 1 election: Requested vote from peers 5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193), e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765)
I20261120 03:03:41.943624  4150 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "c94544a8d7604d15b3a451f3657b06a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5686819a367f4567b741ac52b51c69a3"
I20261120 03:03:41.943830  4225 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "c94544a8d7604d15b3a451f3657b06a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e554bd281f3844c098fb147463c08a84"
I20261120 03:03:41.944176  4150 raft_consensus.cc:3296] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:41.944373  4225 raft_consensus.cc:3296] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:41.945245  4150 raft_consensus.cc:2062] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate c94544a8d7604d15b3a451f3657b06a7 in term 1
I20261120 03:03:41.945323  4225 raft_consensus.cc:2062] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate c94544a8d7604d15b3a451f3657b06a7 in term 1
I20261120 03:03:41.952020  4264 leader_election.cc:304] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c94544a8d7604d15b3a451f3657b06a7, e554bd281f3844c098fb147463c08a84; no voters: 
I20261120 03:03:41.952759  4350 raft_consensus.cc:3024] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:41.953161  4350 raft_consensus.cc:744] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 1 LEADER]: Becoming Leader. State: Replica: c94544a8d7604d15b3a451f3657b06a7, State: Running, Role: LEADER
I20261120 03:03:41.953874  4350 consensus_queue.cc:238] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:41.959921  4354 consensus_queue.cc:1049] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:41.962930  4047 catalog_manager.cc:5856] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67). New cstate: current_term: 1 leader_uuid: "c94544a8d7604d15b3a451f3657b06a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } }
I20261120 03:03:41.966979  4356 consensus_queue.cc:1049] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261120 03:03:42.220103  4140 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:42.222905  4289 tablet_service.cc:1451] Tablet server has 1 leaders and 0 scanners
I20261120 03:03:42.223083  4215 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 7d9d0a04125446849a404f868bc1cf29 | 127.0.96.126:33659 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address      | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
 5686819a367f4567b741ac52b51c69a3 | 127.0.96.65:37193 | HEALTHY | <none>   |       0        |       0
 c94544a8d7604d15b3a451f3657b06a7 | 127.0.96.67:45369 | HEALTHY | <none>   |       1        |       0
 e554bd281f3844c098fb147463c08a84 | 127.0.96.66:45765 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
     Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20261120 03:03:42.443405   385 log_verifier.cc:126] Checking tablet 961865cf1efc4c71a4a12606b68f48d5
I20261120 03:03:42.502789   385 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 961865cf1efc4c71a4a12606b68f48d5
I20261120 03:03:42.503320   385 tablet_replica.cc:333] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: stopping tablet replica
I20261120 03:03:42.503786   385 raft_consensus.cc:2440] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:42.504597   385 raft_consensus.cc:2469] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:42.700480  4140 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:42.700588  4289 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:42.702387  4215 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 7d9d0a04125446849a404f868bc1cf29 | 127.0.96.126:33659 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address      | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
 5686819a367f4567b741ac52b51c69a3 | 127.0.96.65:37193 | HEALTHY | <none>   |       0        |       0
 c94544a8d7604d15b3a451f3657b06a7 | 127.0.96.67:45369 | HEALTHY | <none>   |       0        |       0
 e554bd281f3844c098fb147463c08a84 | 127.0.96.66:45765 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet 961865cf1efc4c71a4a12606b68f48d5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
  5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193): RUNNING
  e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765): RUNNING
All reported replicas are:
  A = c94544a8d7604d15b3a451f3657b06a7
  B = 5686819a367f4567b741ac52b51c69a3
  C = e554bd281f3844c098fb147463c08a84
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A*  B   C              |              |              | Yes
 A             | [config not available] |              |              | 
 B             | A*  B   C              | 1            | -1           | Yes
 C             | A*  B   C              | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20261120 03:03:42.896771   385 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261120 03:03:43.242240  4140 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:43.242638  4289 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:43.246492  4215 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 7d9d0a04125446849a404f868bc1cf29 | 127.0.96.126:33659 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address      | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
 5686819a367f4567b741ac52b51c69a3 | 127.0.96.65:37193 | HEALTHY | <none>   |       0        |       0
 c94544a8d7604d15b3a451f3657b06a7 | 127.0.96.67:45369 | HEALTHY | <none>   |       0        |       0
 e554bd281f3844c098fb147463c08a84 | 127.0.96.66:45765 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet 961865cf1efc4c71a4a12606b68f48d5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
  5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193): RUNNING
  e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765): RUNNING
All reported replicas are:
  A = c94544a8d7604d15b3a451f3657b06a7
  B = 5686819a367f4567b741ac52b51c69a3
  C = e554bd281f3844c098fb147463c08a84
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A*  B   C              |              |              | Yes
 A             | [config not available] |              |              | 
 B             | A*  B   C              | 1            | -1           | Yes
 C             | A*  B   C              | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20261120 03:03:43.440987   385 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261120 03:03:43.866423  4140 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:43.873127  4289 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:43.874809  4215 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:43.881464  4463 raft_consensus.cc:534] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c94544a8d7604d15b3a451f3657b06a7)
I20261120 03:03:43.882016  4463 raft_consensus.cc:556] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:43.932756  4465 raft_consensus.cc:534] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c94544a8d7604d15b3a451f3657b06a7)
I20261120 03:03:43.933027  4463 leader_election.cc:290] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369), 5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193)
I20261120 03:03:43.933378  4465 raft_consensus.cc:556] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:43.954052  4465 leader_election.cc:290] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369), e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765)
I20261120 03:03:43.958839  4149 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "e554bd281f3844c098fb147463c08a84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5686819a367f4567b741ac52b51c69a3" is_pre_election: true
I20261120 03:03:43.961010  4149 raft_consensus.cc:2062] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate e554bd281f3844c098fb147463c08a84 in term 1
I20261120 03:03:43.962296  4190 leader_election.cc:304] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5686819a367f4567b741ac52b51c69a3, e554bd281f3844c098fb147463c08a84; no voters: 
I20261120 03:03:43.963137  4463 raft_consensus.cc:3024] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 03:03:43.963522  4463 raft_consensus.cc:534] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c94544a8d7604d15b3a451f3657b06a7)
I20261120 03:03:43.963860  4463 raft_consensus.cc:3296] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:43.972311  4463 raft_consensus.cc:556] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:43.981005  4225 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "5686819a367f4567b741ac52b51c69a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e554bd281f3844c098fb147463c08a84" is_pre_election: true
I20261120 03:03:43.981769  4225 raft_consensus.cc:2628] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5686819a367f4567b741ac52b51c69a3 in current term 2: Already voted for candidate e554bd281f3844c098fb147463c08a84 in this term.
I20261120 03:03:43.982640  4299 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "5686819a367f4567b741ac52b51c69a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c94544a8d7604d15b3a451f3657b06a7" is_pre_election: true
I20261120 03:03:43.984457  4299 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "e554bd281f3844c098fb147463c08a84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c94544a8d7604d15b3a451f3657b06a7" is_pre_election: true
I20261120 03:03:43.989393  4463 leader_election.cc:290] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [CANDIDATE]: Term 2 election: Requested vote from peers c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369), 5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193)
I20261120 03:03:43.989914  4149 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "e554bd281f3844c098fb147463c08a84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5686819a367f4567b741ac52b51c69a3"
W20261120 03:03:43.990407  4189 leader_election.cc:343] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): Illegal state: cannot vote while shut down
I20261120 03:03:43.992056  4299 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "961865cf1efc4c71a4a12606b68f48d5" candidate_uuid: "e554bd281f3844c098fb147463c08a84" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c94544a8d7604d15b3a451f3657b06a7"
I20261120 03:03:43.990428  4149 raft_consensus.cc:3296] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 1 FOLLOWER]: Advancing to term 2
W20261120 03:03:43.993280  4189 leader_election.cc:343] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): Illegal state: cannot vote while shut down
I20261120 03:03:43.994136  4149 raft_consensus.cc:2062] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate e554bd281f3844c098fb147463c08a84 in term 2
W20261120 03:03:43.994253  4114 leader_election.cc:343] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): Illegal state: cannot vote while shut down
I20261120 03:03:43.999852  4114 leader_election.cc:304] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5686819a367f4567b741ac52b51c69a3; no voters: c94544a8d7604d15b3a451f3657b06a7, e554bd281f3844c098fb147463c08a84
I20261120 03:03:44.002434  4465 raft_consensus.cc:2966] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261120 03:03:44.003104  4190 leader_election.cc:304] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5686819a367f4567b741ac52b51c69a3, e554bd281f3844c098fb147463c08a84; no voters: c94544a8d7604d15b3a451f3657b06a7
I20261120 03:03:44.004276  4463 raft_consensus.cc:3024] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:44.004753  4463 raft_consensus.cc:744] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 LEADER]: Becoming Leader. State: Replica: e554bd281f3844c098fb147463c08a84, State: Running, Role: LEADER
I20261120 03:03:44.005555  4463 consensus_queue.cc:238] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:44.019018  4149 raft_consensus.cc:1338] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Refusing update from remote peer e554bd281f3844c098fb147463c08a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20261120 03:03:44.020713  4470 consensus_queue.cc:1049] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20261120 03:03:44.020612  4048 catalog_manager.cc:5856] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 reported cstate change: term changed from 1 to 2, leader changed from c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67) to e554bd281f3844c098fb147463c08a84 (127.0.96.66). New cstate: current_term: 2 leader_uuid: "e554bd281f3844c098fb147463c08a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } }
W20261120 03:03:44.028003  4189 consensus_peers.cc:597] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 -> Peer c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): Couldn't send request to peer c94544a8d7604d15b3a451f3657b06a7. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 7d9d0a04125446849a404f868bc1cf29 | 127.0.96.126:33659 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address      | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
 5686819a367f4567b741ac52b51c69a3 | 127.0.96.65:37193 | HEALTHY | <none>   |       0        |       0
 c94544a8d7604d15b3a451f3657b06a7 | 127.0.96.67:45369 | HEALTHY | <none>   |       0        |       0
 e554bd281f3844c098fb147463c08a84 | 127.0.96.66:45765 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet 961865cf1efc4c71a4a12606b68f48d5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
  5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193): RUNNING
  e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765): RUNNING
All reported replicas are:
  A = c94544a8d7604d15b3a451f3657b06a7
  B = 5686819a367f4567b741ac52b51c69a3
  C = e554bd281f3844c098fb147463c08a84
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A*  B   C              |              |              | Yes
 A             | [config not available] |              |              | 
 B             | A*  B   C              | 1            | -1           | Yes
 C             | A*  B   C              | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20261120 03:03:44.140944   385 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261120 03:03:44.592993  4463 raft_consensus.cc:1070] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: Attempting to remove follower c94544a8d7604d15b3a451f3657b06a7 from the Raft config. Reason: The tablet replica hosted on peer c94544a8d7604d15b3a451f3657b06a7 has failed
I20261120 03:03:44.595585  4463 consensus_queue.cc:238] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } }
I20261120 03:03:44.598932  4149 raft_consensus.cc:1338] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Refusing update from remote peer e554bd281f3844c098fb147463c08a84: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:44.600278  4488 consensus_queue.cc:1049] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:44.605387  4463 raft_consensus.cc:3183] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } }
I20261120 03:03:44.608626  4149 raft_consensus.cc:3183] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } }
I20261120 03:03:44.611158  4046 catalog_manager.cc:5856] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 reported cstate change: config changed from index -1 to 3, VOTER c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67) evicted. New cstate: current_term: 2 leader_uuid: "e554bd281f3844c098fb147463c08a84" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } }
I20261120 03:03:44.626425  4289 tablet_service.cc:1542] Processing DeleteTablet for tablet 961865cf1efc4c71a4a12606b68f48d5 with delete_type TABLET_DATA_TOMBSTONED (TS c94544a8d7604d15b3a451f3657b06a7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50226
I20261120 03:03:44.628680  4499 ts_tablet_manager.cc:1962] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20261120 03:03:44.628845  4225 consensus_queue.cc:238] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } }
W20261120 03:03:44.634150  4189 consensus_peers.cc:597] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 -> Peer c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): Couldn't send request to peer c94544a8d7604d15b3a451f3657b06a7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:44.634547  4149 raft_consensus.cc:1338] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Refusing update from remote peer e554bd281f3844c098fb147463c08a84: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20261120 03:03:44.635881  4488 consensus_queue.cc:1049] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20261120 03:03:44.641222  4463 raft_consensus.cc:3183] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } }
I20261120 03:03:44.642745  4149 raft_consensus.cc:3183] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } }
I20261120 03:03:44.644281  4034 catalog_manager.cc:5263] ChangeConfig:ADD_PEER:VOTER RPC for tablet 961865cf1efc4c71a4a12606b68f48d5 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20261120 03:03:44.645140  4499 ts_tablet_manager.cc:1975] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20261120 03:03:44.645622  4499 log.cc:1199] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Deleting WAL directory at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1795143768723645-385-0/minicluster-data/ts-2-root/wals/961865cf1efc4c71a4a12606b68f48d5
I20261120 03:03:44.647583  4033 catalog_manager.cc:5081] TS c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): tablet 961865cf1efc4c71a4a12606b68f48d5 (table test-workload [id=74fbc5a4629d4a4aa0d813b8d2120ae4]) successfully deleted
I20261120 03:03:44.649461  4046 catalog_manager.cc:5856] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 reported cstate change: config changed from index 3 to 4, VOTER c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67) added. New cstate: current_term: 2 leader_uuid: "e554bd281f3844c098fb147463c08a84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } } }
I20261120 03:03:44.720041  4140 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:44.729888  4289 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:44.733116  4215 tablet_service.cc:1451] Tablet server has 1 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 7d9d0a04125446849a404f868bc1cf29 | 127.0.96.126:33659 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address      | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
 5686819a367f4567b741ac52b51c69a3 | 127.0.96.65:37193 | HEALTHY | <none>   |       0        |       0
 c94544a8d7604d15b3a451f3657b06a7 | 127.0.96.67:45369 | HEALTHY | <none>   |       0        |       0
 e554bd281f3844c098fb147463c08a84 | 127.0.96.66:45765 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet 961865cf1efc4c71a4a12606b68f48d5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  5686819a367f4567b741ac52b51c69a3 (127.0.96.65:37193): RUNNING
  e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765): RUNNING [LEADER]
  c94544a8d7604d15b3a451f3657b06a7 (127.0.96.67:45369): not running
    State:       FAILED
    Data state:  TABLET_DATA_TOMBSTONED
    Last status: Deleted tablet blocks from disk
All reported replicas are:
  A = 5686819a367f4567b741ac52b51c69a3
  B = e554bd281f3844c098fb147463c08a84
  C = c94544a8d7604d15b3a451f3657b06a7
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A   B*  C              |              |              | Yes
 A             | A   B*  C              | 2            | 4            | Yes
 B             | A   B*  C              | 2            | 4            | Yes
 C             | [config not available] |              |              | 

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20261120 03:03:44.923781   385 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261120 03:03:45.138525  4514 ts_tablet_manager.cc:979] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Initiating tablet copy from peer e554bd281f3844c098fb147463c08a84 (127.0.96.66:45765)
I20261120 03:03:45.139796  4514 tablet_copy_client.cc:323] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.96.66:45765
I20261120 03:03:45.141235  4235 tablet_copy_service.cc:140] P e554bd281f3844c098fb147463c08a84: Received BeginTabletCopySession request for tablet 961865cf1efc4c71a4a12606b68f48d5 from peer c94544a8d7604d15b3a451f3657b06a7 ({username='slave'} at 127.0.0.1:43716)
I20261120 03:03:45.141659  4235 tablet_copy_service.cc:161] P e554bd281f3844c098fb147463c08a84: Beginning new tablet copy session on tablet 961865cf1efc4c71a4a12606b68f48d5 from peer c94544a8d7604d15b3a451f3657b06a7 at {username='slave'} at 127.0.0.1:43716: session id = c94544a8d7604d15b3a451f3657b06a7-961865cf1efc4c71a4a12606b68f48d5
I20261120 03:03:45.146488  4235 tablet_copy_source_session.cc:215] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: Tablet Copy: opened 0 blocks and 1 log segments
I20261120 03:03:45.148399  4514 ts_tablet_manager.cc:1962] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Deleting tablet data with delete state TABLET_DATA_COPYING
I20261120 03:03:45.158772  4514 ts_tablet_manager.cc:1975] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20261120 03:03:45.159324  4514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961865cf1efc4c71a4a12606b68f48d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:45.164237  4514 tablet_copy_client.cc:806] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: tablet copy: Starting download of 0 data blocks...
I20261120 03:03:45.164649  4514 tablet_copy_client.cc:670] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: tablet copy: Starting download of 1 WAL segments...
I20261120 03:03:45.167639  4514 tablet_copy_client.cc:538] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261120 03:03:45.172911  4514 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Bootstrap starting.
I20261120 03:03:45.188017  4514 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 03:03:45.188683  4514 tablet_bootstrap.cc:492] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Bootstrap complete.
I20261120 03:03:45.189182  4514 ts_tablet_manager.cc:1449] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Time spent bootstrapping tablet: real 0.016s	user 0.009s	sys 0.006s
I20261120 03:03:45.190989  4514 raft_consensus.cc:366] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } }
I20261120 03:03:45.191574  4514 raft_consensus.cc:787] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94544a8d7604d15b3a451f3657b06a7, State: Initialized, Role: FOLLOWER
I20261120 03:03:45.192199  4514 consensus_queue.cc:261] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5686819a367f4567b741ac52b51c69a3" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 37193 } } peers { permanent_uuid: "e554bd281f3844c098fb147463c08a84" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 45765 } } peers { permanent_uuid: "c94544a8d7604d15b3a451f3657b06a7" member_type: VOTER last_known_addr { host: "127.0.96.67" port: 45369 } }
I20261120 03:03:45.194197  4514 ts_tablet_manager.cc:1480] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.004s
I20261120 03:03:45.195812  4235 tablet_copy_service.cc:342] P e554bd281f3844c098fb147463c08a84: Request end of tablet copy session c94544a8d7604d15b3a451f3657b06a7-961865cf1efc4c71a4a12606b68f48d5 received from {username='slave'} at 127.0.0.1:43716
I20261120 03:03:45.196193  4235 tablet_copy_service.cc:434] P e554bd281f3844c098fb147463c08a84: ending tablet copy session c94544a8d7604d15b3a451f3657b06a7-961865cf1efc4c71a4a12606b68f48d5 on tablet 961865cf1efc4c71a4a12606b68f48d5 with peer c94544a8d7604d15b3a451f3657b06a7
I20261120 03:03:45.614084  4299 raft_consensus.cc:1279] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.3->[2.4-2.4]   Dedup: 2.4->[]
I20261120 03:03:45.633736  4289 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:45.638548  4140 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:45.640870  4215 tablet_service.cc:1451] Tablet server has 1 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 7d9d0a04125446849a404f868bc1cf29 | 127.0.96.126:33659 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address      | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
 5686819a367f4567b741ac52b51c69a3 | 127.0.96.65:37193 | HEALTHY | <none>   |       0        |       0
 c94544a8d7604d15b3a451f3657b06a7 | 127.0.96.67:45369 | HEALTHY | <none>   |       0        |       0
 e554bd281f3844c098fb147463c08a84 | 127.0.96.66:45765 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
     Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20261120 03:03:45.846527   385 log_verifier.cc:126] Checking tablet 961865cf1efc4c71a4a12606b68f48d5
I20261120 03:03:45.896865   385 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 961865cf1efc4c71a4a12606b68f48d5
I20261120 03:03:45.899945   385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:45.923861   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:45.924510   385 tablet_replica.cc:333] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3: stopping tablet replica
I20261120 03:03:45.925022   385 raft_consensus.cc:2440] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:45.925514   385 raft_consensus.cc:2469] T 961865cf1efc4c71a4a12606b68f48d5 P 5686819a367f4567b741ac52b51c69a3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:45.944460   385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:45.952631   385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:03:45.969669   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:45.970145   385 tablet_replica.cc:333] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84: stopping tablet replica
I20261120 03:03:45.970569   385 raft_consensus.cc:2440] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 LEADER]: Raft consensus shutting down.
I20261120 03:03:45.971345   385 raft_consensus.cc:2469] T 961865cf1efc4c71a4a12606b68f48d5 P e554bd281f3844c098fb147463c08a84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:45.989751   385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:03:45.997961   385 tablet_server.cc:179] TabletServer@127.0.96.67:0 shutting down...
I20261120 03:03:46.013463   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:46.014002   385 tablet_replica.cc:333] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7: stopping tablet replica
I20261120 03:03:46.014456   385 raft_consensus.cc:2440] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:46.014849   385 raft_consensus.cc:2469] T 961865cf1efc4c71a4a12606b68f48d5 P c94544a8d7604d15b3a451f3657b06a7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:46.033376   385 tablet_server.cc:196] TabletServer@127.0.96.67:0 shutdown complete.
I20261120 03:03:46.041604   385 master.cc:585] Master@127.0.96.126:33659 shutting down...
I20261120 03:03:46.054625   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:46.055089   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:46.055446   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7d9d0a04125446849a404f868bc1cf29: stopping tablet replica
I20261120 03:03:46.073941   385 master.cc:607] Master@127.0.96.126:33659 shutdown complete.
[       OK ] FailedTabletsAreReplacedITest.OneReplica/0 (4802 ms)
[ RUN      ] FailedTabletsAreReplacedITest.OneReplica/1
I20261120 03:03:46.099690   385 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.96.126:45111
I20261120 03:03:46.100631   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:46.105304  4549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.106446  4550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.107489  4552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:46.108047   385 server_base.cc:1061] running on GCE node
I20261120 03:03:46.108997   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:46.109228   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:46.109400   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143826109369 us; error 0 us; skew 500 ppm
I20261120 03:03:46.109911   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:46.111814   385 webserver.cc:577] Webserver started at http://127.0.96.126:43999/ using document root <none> and password file <none>
I20261120 03:03:46.112322   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:46.112505   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:46.112787   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:46.113786   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/master-0-root/instance:
uuid: "ab901ce6ae784c0997186711f9a199c2"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.117836   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.003s
I20261120 03:03:46.120779  4557 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:46.121517   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20261120 03:03:46.121783   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/master-0-root
uuid: "ab901ce6ae784c0997186711f9a199c2"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.122025   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:46.137247   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:46.138085   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:46.169865   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.126:45111
I20261120 03:03:46.169950  4608 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.126:45111 every 8 connection(s)
I20261120 03:03:46.173592  4609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:46.183063  4609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2: Bootstrap starting.
I20261120 03:03:46.187094  4609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:46.190445  4609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2: No bootstrap required, opened a new log
I20261120 03:03:46.192219  4609 raft_consensus.cc:366] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab901ce6ae784c0997186711f9a199c2" member_type: VOTER }
I20261120 03:03:46.192544  4609 raft_consensus.cc:392] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:46.192713  4609 raft_consensus.cc:787] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab901ce6ae784c0997186711f9a199c2, State: Initialized, Role: FOLLOWER
I20261120 03:03:46.193161  4609 consensus_queue.cc:261] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab901ce6ae784c0997186711f9a199c2" member_type: VOTER }
I20261120 03:03:46.193519  4609 raft_consensus.cc:407] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 03:03:46.193704  4609 raft_consensus.cc:534] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 03:03:46.193900  4609 raft_consensus.cc:3296] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:46.194837  4609 raft_consensus.cc:556] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab901ce6ae784c0997186711f9a199c2" member_type: VOTER }
I20261120 03:03:46.198951  4609 leader_election.cc:304] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab901ce6ae784c0997186711f9a199c2; no voters: 
I20261120 03:03:46.199404  4609 leader_election.cc:290] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20261120 03:03:46.199508  4612 raft_consensus.cc:3024] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:46.200698  4612 raft_consensus.cc:744] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 1 LEADER]: Becoming Leader. State: Replica: ab901ce6ae784c0997186711f9a199c2, State: Running, Role: LEADER
I20261120 03:03:46.201313  4612 consensus_queue.cc:238] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab901ce6ae784c0997186711f9a199c2" member_type: VOTER }
I20261120 03:03:46.202098  4609 sys_catalog.cc:565] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 03:03:46.203719  4614 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab901ce6ae784c0997186711f9a199c2. Latest consensus state: current_term: 1 leader_uuid: "ab901ce6ae784c0997186711f9a199c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab901ce6ae784c0997186711f9a199c2" member_type: VOTER } }
I20261120 03:03:46.204461  4614 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:46.203714  4613 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab901ce6ae784c0997186711f9a199c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab901ce6ae784c0997186711f9a199c2" member_type: VOTER } }
I20261120 03:03:46.205269  4613 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [sys.catalog]: This master's current role is: LEADER
I20261120 03:03:46.206921  4616 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 03:03:46.211404  4616 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 03:03:46.215711   385 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261120 03:03:46.219229  4616 catalog_manager.cc:1398] Generated new cluster ID: 7f33f497200542aeb0de479e3371f4c8
I20261120 03:03:46.219445  4616 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 03:03:46.240929  4616 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 03:03:46.241993  4616 catalog_manager.cc:1555] Loading token signing keys...
I20261120 03:03:46.278982  4616 catalog_manager.cc:6237] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2: Generated new TSK 0
I20261120 03:03:46.279484  4616 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 03:03:46.346292   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:46.351475  4630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.352512  4631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:46.355122   385 server_base.cc:1061] running on GCE node
W20261120 03:03:46.355435  4633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:46.356325   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:46.356518   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:46.356652   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143826356641 us; error 0 us; skew 500 ppm
I20261120 03:03:46.357093   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:46.358989   385 webserver.cc:577] Webserver started at http://127.0.96.65:38183/ using document root <none> and password file <none>
I20261120 03:03:46.359369   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:46.359506   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:46.359691   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:46.360689   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-0-root/instance:
uuid: "4013678e60394190a276ac0dbff7464b"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.364553   385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20261120 03:03:46.367406  4638 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:46.368072   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20261120 03:03:46.368395   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-0-root
uuid: "4013678e60394190a276ac0dbff7464b"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.368646   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:46.378721   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:46.379662   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:46.380973   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:46.382928   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:46.383136   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:46.383344   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:46.383500   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:46.418754   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.65:40039
I20261120 03:03:46.418843  4700 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.65:40039 every 8 connection(s)
I20261120 03:03:46.423411   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:46.429674  4705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.431241  4706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.432718  4708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:46.433269   385 server_base.cc:1061] running on GCE node
I20261120 03:03:46.434403   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:46.434674   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:46.434909   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143826434890 us; error 0 us; skew 500 ppm
I20261120 03:03:46.435056  4701 heartbeater.cc:345] Connected to a master server at 127.0.96.126:45111
I20261120 03:03:46.435346  4701 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:46.435487   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:46.436077  4701 heartbeater.cc:508] Master 127.0.96.126:45111 requested a full tablet report, sending...
I20261120 03:03:46.438010   385 webserver.cc:577] Webserver started at http://127.0.96.66:44291/ using document root <none> and password file <none>
I20261120 03:03:46.438421  4574 ts_manager.cc:210] Registered new tserver with Master: 4013678e60394190a276ac0dbff7464b (127.0.96.65:40039)
I20261120 03:03:46.438653   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:46.438913   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:46.439213   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:46.440573   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-1-root/instance:
uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.440660  4574 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36936
I20261120 03:03:46.444908   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.003s
I20261120 03:03:46.447921  4713 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:46.448626   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20261120 03:03:46.448899   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-1-root
uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.449146   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:46.468667   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:46.469552   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:46.470955   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:46.473026   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:46.473186   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:46.473457   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:46.473608   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:46.510665   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.66:34073
I20261120 03:03:46.510833  4775 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.66:34073 every 8 connection(s)
I20261120 03:03:46.515223   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:46.522864  4779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.523957  4780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.526901  4782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:46.527424  4776 heartbeater.cc:345] Connected to a master server at 127.0.96.126:45111
I20261120 03:03:46.527729   385 server_base.cc:1061] running on GCE node
I20261120 03:03:46.527851  4776 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:46.528656   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20261120 03:03:46.528703  4776 heartbeater.cc:508] Master 127.0.96.126:45111 requested a full tablet report, sending...
W20261120 03:03:46.528980   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:46.529259   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143826529241 us; error 0 us; skew 500 ppm
I20261120 03:03:46.529834   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:46.531131  4574 ts_manager.cc:210] Registered new tserver with Master: eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073)
I20261120 03:03:46.532344   385 webserver.cc:577] Webserver started at http://127.0.96.67:40297/ using document root <none> and password file <none>
I20261120 03:03:46.532967   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:46.532907  4574 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36942
I20261120 03:03:46.533360   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:46.533694   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:46.535100   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-2-root/instance:
uuid: "e68cec25598e47e3ab33cac75ae305da"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.539466   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.007s	sys 0.000s
I20261120 03:03:46.542544  4787 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:46.543314   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20261120 03:03:46.543637   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-2-root
uuid: "e68cec25598e47e3ab33cac75ae305da"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.543902   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:46.559372   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:46.560307   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:46.561553   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:46.563509   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:46.563724   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:46.563954   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:46.564147   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:46.599982   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.67:44387
I20261120 03:03:46.600071  4849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.67:44387 every 8 connection(s)
I20261120 03:03:46.604602   385 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 03:03:46.612017  4853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 03:03:46.613539  4854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:46.614902   385 server_base.cc:1061] running on GCE node
I20261120 03:03:46.614944  4850 heartbeater.cc:345] Connected to a master server at 127.0.96.126:45111
I20261120 03:03:46.615617  4850 heartbeater.cc:462] Registering TS with master...
W20261120 03:03:46.615926  4857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 03:03:46.616460  4850 heartbeater.cc:508] Master 127.0.96.126:45111 requested a full tablet report, sending...
I20261120 03:03:46.616858   385 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261120 03:03:46.617084   385 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261120 03:03:46.617261   385 hybrid_clock.cc:648] HybridClock initialized: now 1795143826617249 us; error 0 us; skew 500 ppm
I20261120 03:03:46.617911   385 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261120 03:03:46.618525  4574 ts_manager.cc:210] Registered new tserver with Master: e68cec25598e47e3ab33cac75ae305da (127.0.96.67:44387)
I20261120 03:03:46.620026  4574 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36948
I20261120 03:03:46.620450   385 webserver.cc:577] Webserver started at http://127.0.96.68:41151/ using document root <none> and password file <none>
I20261120 03:03:46.621025   385 fs_manager.cc:362] Metadata directory not provided
I20261120 03:03:46.621268   385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 03:03:46.621591   385 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 03:03:46.622699   385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-3-root/instance:
uuid: "59b78942986249eda08d27d6863395d8"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.626894   385 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20261120 03:03:46.629859  4861 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:46.630571   385 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20261120 03:03:46.630867   385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-3-root
uuid: "59b78942986249eda08d27d6863395d8"
format_stamp: "Formatted at 2026-11-20 03:03:46 on dist-test-slave-54pf"
I20261120 03:03:46.631152   385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 03:03:46.646580   385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20261120 03:03:46.647521   385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 03:03:46.648828   385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 03:03:46.650867   385 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 03:03:46.651149   385 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:46.651387   385 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 03:03:46.651559   385 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20261120 03:03:46.687908   385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.68:36041
I20261120 03:03:46.688004  4923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.96.68:36041 every 8 connection(s)
I20261120 03:03:46.699859  4924 heartbeater.cc:345] Connected to a master server at 127.0.96.126:45111
I20261120 03:03:46.700229  4924 heartbeater.cc:462] Registering TS with master...
I20261120 03:03:46.700835  4924 heartbeater.cc:508] Master 127.0.96.126:45111 requested a full tablet report, sending...
I20261120 03:03:46.702544  4574 ts_manager.cc:210] Registered new tserver with Master: 59b78942986249eda08d27d6863395d8 (127.0.96.68:36041)
I20261120 03:03:46.702957   385 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.012188357s
I20261120 03:03:46.704056  4574 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36952
I20261120 03:03:46.739713  4574 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36976:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20261120 03:03:46.777551  4889 tablet_service.cc:1495] Processing CreateTablet for tablet d195a20f388941cdb304ec869af88f73 (DEFAULT_TABLE table=test-workload [id=d13eb37246cc49babff26dd8a4d21252]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:46.778615  4889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d195a20f388941cdb304ec869af88f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:46.779963  4666 tablet_service.cc:1495] Processing CreateTablet for tablet d195a20f388941cdb304ec869af88f73 (DEFAULT_TABLE table=test-workload [id=d13eb37246cc49babff26dd8a4d21252]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:46.781162  4666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d195a20f388941cdb304ec869af88f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:46.781445  4741 tablet_service.cc:1495] Processing CreateTablet for tablet d195a20f388941cdb304ec869af88f73 (DEFAULT_TABLE table=test-workload [id=d13eb37246cc49babff26dd8a4d21252]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 03:03:46.782631  4741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d195a20f388941cdb304ec869af88f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 03:03:46.791404  4944 tablet_bootstrap.cc:492] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8: Bootstrap starting.
I20261120 03:03:46.799377  4944 tablet_bootstrap.cc:654] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:46.799803  4946 tablet_bootstrap.cc:492] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b: Bootstrap starting.
I20261120 03:03:46.801764  4945 tablet_bootstrap.cc:492] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: Bootstrap starting.
I20261120 03:03:46.804790  4944 tablet_bootstrap.cc:492] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8: No bootstrap required, opened a new log
I20261120 03:03:46.805271  4944 ts_tablet_manager.cc:1449] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.005s
I20261120 03:03:46.807211  4946 tablet_bootstrap.cc:654] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:46.807979  4945 tablet_bootstrap.cc:654] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: Neither blocks nor log segments found. Creating new log.
I20261120 03:03:46.808087  4944 raft_consensus.cc:366] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:46.808849  4944 raft_consensus.cc:392] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:46.809154  4944 raft_consensus.cc:787] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b78942986249eda08d27d6863395d8, State: Initialized, Role: FOLLOWER
I20261120 03:03:46.809904  4944 consensus_queue.cc:261] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:46.812773  4924 heartbeater.cc:500] Master 127.0.96.126:45111 was elected leader, sending a full tablet report...
I20261120 03:03:46.813289  4944 ts_tablet_manager.cc:1480] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20261120 03:03:46.813407  4946 tablet_bootstrap.cc:492] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b: No bootstrap required, opened a new log
I20261120 03:03:46.813866  4946 ts_tablet_manager.cc:1449] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b: Time spent bootstrapping tablet: real 0.014s	user 0.011s	sys 0.000s
I20261120 03:03:46.814347  4945 tablet_bootstrap.cc:492] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: No bootstrap required, opened a new log
I20261120 03:03:46.814772  4945 ts_tablet_manager.cc:1449] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: Time spent bootstrapping tablet: real 0.013s	user 0.011s	sys 0.000s
I20261120 03:03:46.816360  4946 raft_consensus.cc:366] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:46.816911  4946 raft_consensus.cc:392] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:46.817147  4946 raft_consensus.cc:787] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4013678e60394190a276ac0dbff7464b, State: Initialized, Role: FOLLOWER
I20261120 03:03:46.817539  4945 raft_consensus.cc:366] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:46.817770  4946 consensus_queue.cc:261] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:46.818325  4945 raft_consensus.cc:392] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 03:03:46.818696  4945 raft_consensus.cc:787] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeee66a4d3b24e3a8c35ce5ab79e96ac, State: Initialized, Role: FOLLOWER
I20261120 03:03:46.819495  4945 consensus_queue.cc:261] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:46.822103  4776 heartbeater.cc:500] Master 127.0.96.126:45111 was elected leader, sending a full tablet report...
I20261120 03:03:46.822548  4945 ts_tablet_manager.cc:1480] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20261120 03:03:46.824457  4701 heartbeater.cc:500] Master 127.0.96.126:45111 was elected leader, sending a full tablet report...
I20261120 03:03:46.825717  4946 ts_tablet_manager.cc:1480] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.005s
I20261120 03:03:47.009091  4952 raft_consensus.cc:534] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 03:03:47.009632  4952 raft_consensus.cc:556] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:47.012039  4952 leader_election.cc:290] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59b78942986249eda08d27d6863395d8 (127.0.96.68:36041), 4013678e60394190a276ac0dbff7464b (127.0.96.65:40039)
I20261120 03:03:47.020725  4899 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b78942986249eda08d27d6863395d8" is_pre_election: true
I20261120 03:03:47.021443  4899 raft_consensus.cc:2062] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate eeee66a4d3b24e3a8c35ce5ab79e96ac in term 0
I20261120 03:03:47.021900  4676 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4013678e60394190a276ac0dbff7464b" is_pre_election: true
I20261120 03:03:47.022379  4717 leader_election.cc:304] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59b78942986249eda08d27d6863395d8, eeee66a4d3b24e3a8c35ce5ab79e96ac; no voters: 
I20261120 03:03:47.022612  4676 raft_consensus.cc:2062] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate eeee66a4d3b24e3a8c35ce5ab79e96ac in term 0
I20261120 03:03:47.023064  4952 raft_consensus.cc:3024] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 03:03:47.023484  4952 raft_consensus.cc:534] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 03:03:47.023818  4952 raft_consensus.cc:3296] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:47.024899  4952 raft_consensus.cc:556] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:47.030076  4952 leader_election.cc:290] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [CANDIDATE]: Term 1 election: Requested vote from peers 59b78942986249eda08d27d6863395d8 (127.0.96.68:36041), 4013678e60394190a276ac0dbff7464b (127.0.96.65:40039)
I20261120 03:03:47.030712  4899 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b78942986249eda08d27d6863395d8"
I20261120 03:03:47.031041  4676 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4013678e60394190a276ac0dbff7464b"
I20261120 03:03:47.031157  4899 raft_consensus.cc:3296] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:47.031530  4676 raft_consensus.cc:3296] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 0 FOLLOWER]: Advancing to term 1
I20261120 03:03:47.032027  4899 raft_consensus.cc:2062] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate eeee66a4d3b24e3a8c35ce5ab79e96ac in term 1
I20261120 03:03:47.032428  4676 raft_consensus.cc:2062] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate eeee66a4d3b24e3a8c35ce5ab79e96ac in term 1
I20261120 03:03:47.036778  4717 leader_election.cc:304] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59b78942986249eda08d27d6863395d8, eeee66a4d3b24e3a8c35ce5ab79e96ac; no voters: 
I20261120 03:03:47.037560  4952 raft_consensus.cc:3024] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 1 FOLLOWER]: Leader election won for term 1
I20261120 03:03:47.037981  4952 raft_consensus.cc:744] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 1 LEADER]: Becoming Leader. State: Replica: eeee66a4d3b24e3a8c35ce5ab79e96ac, State: Running, Role: LEADER
I20261120 03:03:47.038738  4952 consensus_queue.cc:238] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:47.044446  4952 consensus_queue.cc:1049] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:47.045560  4956 consensus_queue.cc:1049] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 03:03:47.045956  4573 catalog_manager.cc:5856] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac reported cstate change: term changed from 0 to 1, leader changed from <none> to eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66). New cstate: current_term: 1 leader_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:47.296648  4666 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:47.301026  4889 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:47.309549  4741 tablet_service.cc:1451] Tablet server has 1 leaders and 0 scanners
I20261120 03:03:47.311543  4815 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 ab901ce6ae784c0997186711f9a199c2 | 127.0.96.126:45111 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address      | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
 4013678e60394190a276ac0dbff7464b | 127.0.96.65:40039 | HEALTHY | <none>   |       0        |       0
 59b78942986249eda08d27d6863395d8 | 127.0.96.68:36041 | HEALTHY | <none>   |       0        |       0
 e68cec25598e47e3ab33cac75ae305da | 127.0.96.67:44387 | HEALTHY | <none>   |       0        |       0
 eeee66a4d3b24e3a8c35ce5ab79e96ac | 127.0.96.66:34073 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
     Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |       Host        | Replica Count
-------+----------------------------------+-------------------+---------------
 Small | e68cec25598e47e3ab33cac75ae305da | 127.0.96.67:44387 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20261120 03:03:47.579432   385 log_verifier.cc:126] Checking tablet d195a20f388941cdb304ec869af88f73
I20261120 03:03:47.622825  4850 heartbeater.cc:500] Master 127.0.96.126:45111 was elected leader, sending a full tablet report...
I20261120 03:03:47.636984   385 log_verifier.cc:177] Verified matching terms for 1 ops in tablet d195a20f388941cdb304ec869af88f73
I20261120 03:03:47.638901   385 tablet_replica.cc:333] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: stopping tablet replica
I20261120 03:03:47.639436   385 raft_consensus.cc:2440] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:47.640311   385 raft_consensus.cc:2469] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:47.860884  4889 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:47.863045  4666 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:47.863466  4741 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:47.868448  4815 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 ab901ce6ae784c0997186711f9a199c2 | 127.0.96.126:45111 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address      | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
 4013678e60394190a276ac0dbff7464b | 127.0.96.65:40039 | HEALTHY | <none>   |       0        |       0
 59b78942986249eda08d27d6863395d8 | 127.0.96.68:36041 | HEALTHY | <none>   |       0        |       0
 e68cec25598e47e3ab33cac75ae305da | 127.0.96.67:44387 | HEALTHY | <none>   |       0        |       0
 eeee66a4d3b24e3a8c35ce5ab79e96ac | 127.0.96.66:34073 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
Tablet d195a20f388941cdb304ec869af88f73 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  59b78942986249eda08d27d6863395d8 (127.0.96.68:36041): RUNNING
  4013678e60394190a276ac0dbff7464b (127.0.96.65:40039): RUNNING
  eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
  A = 59b78942986249eda08d27d6863395d8
  B = 4013678e60394190a276ac0dbff7464b
  C = eeee66a4d3b24e3a8c35ce5ab79e96ac
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A   B   C*             |              |              | Yes
 A             | A   B   C*             | 1            | -1           | Yes
 B             | A   B   C*             | 1            | -1           | Yes
 C             | [config not available] |              |              | 

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |       Host        | Replica Count
-------+----------------------------------+-------------------+---------------
 Small | e68cec25598e47e3ab33cac75ae305da | 127.0.96.67:44387 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20261120 03:03:48.099592   385 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261120 03:03:48.459100  4666 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:48.465679  4889 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:48.467797  4815 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:48.469378  4741 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:48.568826  5050 raft_consensus.cc:534] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeee66a4d3b24e3a8c35ce5ab79e96ac)
I20261120 03:03:48.578768  5050 raft_consensus.cc:556] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:48.600183  5050 leader_election.cc:290] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4013678e60394190a276ac0dbff7464b (127.0.96.65:40039), eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073)
I20261120 03:03:48.629886  4751 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "59b78942986249eda08d27d6863395d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" is_pre_election: true
W20261120 03:03:48.631368  4863 leader_election.cc:343] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073): Illegal state: cannot vote while shut down
I20261120 03:03:48.631697  4676 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "59b78942986249eda08d27d6863395d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4013678e60394190a276ac0dbff7464b" is_pre_election: true
I20261120 03:03:48.633631  4864 leader_election.cc:304] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59b78942986249eda08d27d6863395d8; no voters: 4013678e60394190a276ac0dbff7464b, eeee66a4d3b24e3a8c35ce5ab79e96ac
I20261120 03:03:48.634711  5050 raft_consensus.cc:2966] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 ab901ce6ae784c0997186711f9a199c2 | 127.0.96.126:45111 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address      | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
 4013678e60394190a276ac0dbff7464b | 127.0.96.65:40039 | HEALTHY | <none>   |       0        |       0
 59b78942986249eda08d27d6863395d8 | 127.0.96.68:36041 | HEALTHY | <none>   |       0        |       0
 e68cec25598e47e3ab33cac75ae305da | 127.0.96.67:44387 | HEALTHY | <none>   |       0        |       0
 eeee66a4d3b24e3a8c35ce5ab79e96ac | 127.0.96.66:34073 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
Tablet d195a20f388941cdb304ec869af88f73 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  59b78942986249eda08d27d6863395d8 (127.0.96.68:36041): RUNNING
  4013678e60394190a276ac0dbff7464b (127.0.96.65:40039): RUNNING
  eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
  A = 59b78942986249eda08d27d6863395d8
  B = 4013678e60394190a276ac0dbff7464b
  C = eeee66a4d3b24e3a8c35ce5ab79e96ac
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A   B   C*             |              |              | Yes
 A             | A   B   C*             | 1            | -1           | Yes
 B             | A   B   C*             | 1            | -1           | Yes
 C             | [config not available] |              |              | 

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |       Host        | Replica Count
-------+----------------------------------+-------------------+---------------
 Small | e68cec25598e47e3ab33cac75ae305da | 127.0.96.67:44387 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20261120 03:03:48.717234   385 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20261120 03:03:49.031080  5060 raft_consensus.cc:534] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eeee66a4d3b24e3a8c35ce5ab79e96ac)
I20261120 03:03:49.031570  5060 raft_consensus.cc:556] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:49.033778  5060 leader_election.cc:290] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59b78942986249eda08d27d6863395d8 (127.0.96.68:36041), eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073)
I20261120 03:03:49.044205  4899 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "4013678e60394190a276ac0dbff7464b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59b78942986249eda08d27d6863395d8" is_pre_election: true
I20261120 03:03:49.044996  4899 raft_consensus.cc:2062] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 4013678e60394190a276ac0dbff7464b in term 1
I20261120 03:03:49.046286  4642 leader_election.cc:304] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4013678e60394190a276ac0dbff7464b, 59b78942986249eda08d27d6863395d8; no voters: 
I20261120 03:03:49.047107  5060 raft_consensus.cc:3024] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 03:03:49.047014  4751 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "4013678e60394190a276ac0dbff7464b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" is_pre_election: true
I20261120 03:03:49.048115  5060 raft_consensus.cc:534] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 1 FOLLOWER]: Starting leader election (detected failure of leader eeee66a4d3b24e3a8c35ce5ab79e96ac)
W20261120 03:03:49.048427  4640 leader_election.cc:343] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073): Illegal state: cannot vote while shut down
I20261120 03:03:49.048599  5060 raft_consensus.cc:3296] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 1 FOLLOWER]: Advancing to term 2
I20261120 03:03:49.050215  5060 raft_consensus.cc:556] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:49.057989  5060 leader_election.cc:290] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [CANDIDATE]: Term 2 election: Requested vote from peers 59b78942986249eda08d27d6863395d8 (127.0.96.68:36041), eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073)
I20261120 03:03:49.058753  4899 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "4013678e60394190a276ac0dbff7464b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59b78942986249eda08d27d6863395d8"
I20261120 03:03:49.059052  4751 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "d195a20f388941cdb304ec869af88f73" candidate_uuid: "4013678e60394190a276ac0dbff7464b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac"
I20261120 03:03:49.059221  4899 raft_consensus.cc:3296] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 1 FOLLOWER]: Advancing to term 2
W20261120 03:03:49.060195  4640 leader_election.cc:343] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073): Illegal state: cannot vote while shut down
I20261120 03:03:49.063990  4899 raft_consensus.cc:2062] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate 4013678e60394190a276ac0dbff7464b in term 2
I20261120 03:03:49.064875  4642 leader_election.cc:304] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4013678e60394190a276ac0dbff7464b, 59b78942986249eda08d27d6863395d8; no voters: eeee66a4d3b24e3a8c35ce5ab79e96ac
I20261120 03:03:49.065515  5060 raft_consensus.cc:3024] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 2 FOLLOWER]: Leader election won for term 2
I20261120 03:03:49.065984  5060 raft_consensus.cc:744] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 2 LEADER]: Becoming Leader. State: Replica: 4013678e60394190a276ac0dbff7464b, State: Running, Role: LEADER
I20261120 03:03:49.066710  5060 consensus_queue.cc:238] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } }
I20261120 03:03:49.073774  4574 catalog_manager.cc:5856] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b reported cstate change: term changed from 1 to 2, leader changed from eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66) to 4013678e60394190a276ac0dbff7464b (127.0.96.65). New cstate: current_term: 2 leader_uuid: "4013678e60394190a276ac0dbff7464b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeee66a4d3b24e3a8c35ce5ab79e96ac" member_type: VOTER last_known_addr { host: "127.0.96.66" port: 34073 } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:49.077309  4899 raft_consensus.cc:1338] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Refusing update from remote peer 4013678e60394190a276ac0dbff7464b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20261120 03:03:49.078903  5064 consensus_queue.cc:1049] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261120 03:03:49.080024  4640 consensus_queue.cc:580] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [LEADER]: The tablet replica hosted on peer eeee66a4d3b24e3a8c35ce5ab79e96ac has failed
W20261120 03:03:49.080476  4640 consensus_peers.cc:597] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b -> Peer eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073): Couldn't send request to peer eeee66a4d3b24e3a8c35ce5ab79e96ac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:49.118326  4676 consensus_queue.cc:238] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } }
I20261120 03:03:49.122988  4899 raft_consensus.cc:1338] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Refusing update from remote peer 4013678e60394190a276ac0dbff7464b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20261120 03:03:49.124249  5060 consensus_queue.cc:1049] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 03:03:49.129558  5066 raft_consensus.cc:3183] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } }
I20261120 03:03:49.132598  4560 catalog_manager.cc:5263] ChangeConfig:REMOVE_PEER RPC for tablet d195a20f388941cdb304ec869af88f73 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20261120 03:03:49.136091  4573 catalog_manager.cc:5856] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b reported cstate change: config changed from index -1 to 3, VOTER eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66) evicted. New cstate: current_term: 2 leader_uuid: "4013678e60394190a276ac0dbff7464b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } health_report { overall_health: HEALTHY } } }
I20261120 03:03:49.138659  4899 raft_consensus.cc:3183] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } }
I20261120 03:03:49.152062  4741 tablet_service.cc:1542] Processing DeleteTablet for tablet d195a20f388941cdb304ec869af88f73 with delete_type TABLET_DATA_TOMBSTONED (TS eeee66a4d3b24e3a8c35ce5ab79e96ac not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43146
I20261120 03:03:49.153717  5086 ts_tablet_manager.cc:1962] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20261120 03:03:49.156804  4676 consensus_queue.cc:238] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "e68cec25598e47e3ab33cac75ae305da" member_type: NON_VOTER last_known_addr { host: "127.0.96.67" port: 44387 } attrs { promote: true } }
I20261120 03:03:49.165867  4899 raft_consensus.cc:1338] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Refusing update from remote peer 4013678e60394190a276ac0dbff7464b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20261120 03:03:49.166860  5086 ts_tablet_manager.cc:1975] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20261120 03:03:49.167259  5086 log.cc:1199] T d195a20f388941cdb304ec869af88f73 P eeee66a4d3b24e3a8c35ce5ab79e96ac: Deleting WAL directory at /tmp/dist-test-taskZ2Ac__/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1795143768723645-385-0/minicluster-data/ts-1-root/wals/d195a20f388941cdb304ec869af88f73
I20261120 03:03:49.167615  5064 consensus_queue.cc:1049] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20261120 03:03:49.169291  4559 catalog_manager.cc:5081] TS eeee66a4d3b24e3a8c35ce5ab79e96ac (127.0.96.66:34073): tablet d195a20f388941cdb304ec869af88f73 (table test-workload [id=d13eb37246cc49babff26dd8a4d21252]) successfully deleted
I20261120 03:03:49.173926  5066 raft_consensus.cc:3183] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER e68cec25598e47e3ab33cac75ae305da (127.0.96.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "e68cec25598e47e3ab33cac75ae305da" member_type: NON_VOTER last_known_addr { host: "127.0.96.67" port: 44387 } attrs { promote: true } } }
I20261120 03:03:49.176324  4560 catalog_manager.cc:5263] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d195a20f388941cdb304ec869af88f73 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20261120 03:03:49.175645  4899 raft_consensus.cc:3183] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER e68cec25598e47e3ab33cac75ae305da (127.0.96.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } } peers { permanent_uuid: "e68cec25598e47e3ab33cac75ae305da" member_type: NON_VOTER last_known_addr { host: "127.0.96.67" port: 44387 } attrs { promote: true } } }
W20261120 03:03:49.180035  4641 consensus_peers.cc:597] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b -> Peer e68cec25598e47e3ab33cac75ae305da (127.0.96.67:44387): Couldn't send request to peer e68cec25598e47e3ab33cac75ae305da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d195a20f388941cdb304ec869af88f73. This is attempt 1: this message will repeat every 5th retry.
I20261120 03:03:49.181865  4574 catalog_manager.cc:5856] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b reported cstate change: config changed from index 3 to 4, NON_VOTER e68cec25598e47e3ab33cac75ae305da (127.0.96.67) added. New cstate: current_term: 2 leader_uuid: "4013678e60394190a276ac0dbff7464b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59b78942986249eda08d27d6863395d8" member_type: VOTER last_known_addr { host: "127.0.96.68" port: 36041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4013678e60394190a276ac0dbff7464b" member_type: VOTER last_known_addr { host: "127.0.96.65" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e68cec25598e47e3ab33cac75ae305da" member_type: NON_VOTER last_known_addr { host: "127.0.96.67" port: 44387 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20261120 03:03:49.224613  4666 tablet_service.cc:1451] Tablet server has 1 leaders and 0 scanners
I20261120 03:03:49.229179  4889 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:49.236581  4815 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
I20261120 03:03:49.238199  4741 tablet_service.cc:1451] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 ab901ce6ae784c0997186711f9a199c2 | 127.0.96.126:45111 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address      | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
 4013678e60394190a276ac0dbff7464b | 127.0.96.65:40039 | HEALTHY | <none>   |       1        |       0
 59b78942986249eda08d27d6863395d8 | 127.0.96.68:36041 | HEALTHY | <none>   |       0        |       0
 e68cec25598e47e3ab33cac75ae305da | 127.0.96.67:44387 | HEALTHY | <none>   |       0        |       0
 eeee66a4d3b24e3a8c35ce5ab79e96ac | 127.0.96.66:34073 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
     Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |       Host        | Replica Count
-------+----------------------------------+-------------------+---------------
 Small | eeee66a4d3b24e3a8c35ce5ab79e96ac | 127.0.96.66:34073 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20261120 03:03:49.551662   385 log_verifier.cc:126] Checking tablet d195a20f388941cdb304ec869af88f73
I20261120 03:03:49.587256   385 log_verifier.cc:177] Verified matching terms for 4 ops in tablet d195a20f388941cdb304ec869af88f73
I20261120 03:03:49.591552   385 tablet_server.cc:179] TabletServer@127.0.96.65:0 shutting down...
I20261120 03:03:49.615511   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:49.616271   385 tablet_replica.cc:333] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b: stopping tablet replica
I20261120 03:03:49.616806   385 raft_consensus.cc:2440] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 2 LEADER]: Raft consensus shutting down.
I20261120 03:03:49.617897   385 raft_consensus.cc:2469] T d195a20f388941cdb304ec869af88f73 P 4013678e60394190a276ac0dbff7464b [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:49.630429   385 tablet_server.cc:196] TabletServer@127.0.96.65:0 shutdown complete.
I20261120 03:03:49.644192   385 tablet_server.cc:179] TabletServer@127.0.96.66:0 shutting down...
I20261120 03:03:49.663689   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:49.681990   385 tablet_server.cc:196] TabletServer@127.0.96.66:0 shutdown complete.
I20261120 03:03:49.693478   385 tablet_server.cc:179] TabletServer@127.0.96.67:0 shutting down...
I20261120 03:03:49.714630   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:49.730579   385 tablet_server.cc:196] TabletServer@127.0.96.67:0 shutdown complete.
I20261120 03:03:49.739852   385 tablet_server.cc:179] TabletServer@127.0.96.68:0 shutting down...
I20261120 03:03:49.761176   385 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261120 03:03:49.761825   385 tablet_replica.cc:333] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8: stopping tablet replica
I20261120 03:03:49.762534   385 raft_consensus.cc:2440] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20261120 03:03:49.763254   385 raft_consensus.cc:2469] T d195a20f388941cdb304ec869af88f73 P 59b78942986249eda08d27d6863395d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:49.783582   385 tablet_server.cc:196] TabletServer@127.0.96.68:0 shutdown complete.
I20261120 03:03:49.794668   385 master.cc:585] Master@127.0.96.126:45111 shutting down...
I20261120 03:03:49.814589   385 raft_consensus.cc:2440] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 1 LEADER]: Raft consensus shutting down.
I20261120 03:03:49.815093   385 raft_consensus.cc:2469] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 03:03:49.815389   385 tablet_replica.cc:333] T 00000000000000000000000000000000 P ab901ce6ae784c0997186711f9a199c2: stopping tablet replica
I20261120 03:03:49.833451   385 master.cc:607] Master@127.0.96.126:45111 shutdown complete.
[       OK ] FailedTabletsAreReplacedITest.OneReplica/1 (3764 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (8566 ms total)

[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (60981 ms total)
[  PASSED  ] 11 tests.
[  SKIPPED ] 1 test, listed below:
[  SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
[  FAILED  ] 1 test, listed below:
[  FAILED  ] TsTabletManagerITest.TestTableStats

 1 FAILED TEST
I20261120 03:03:49.872270   385 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~9 seconds ago
I20261120 03:03:49.872552   385 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 6 messages since previous log ~13 seconds ago
I20261120 03:03:49.872710   385 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:648: suppressed but not reported on 72 messages since previous log ~56 seconds ago
I20261120 03:03:49.872943   385 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2554: suppressed but not reported on 7 messages since previous log ~12 seconds ago