TsTabletManagerITest.TestTableStats: /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/ts_tablet_manager-itest.cc:786: Failure
Expected equality of these values:
  live_row_count
    Which is: 795
  table_info->GetMetrics()->live_row_count->value()
    Which is: 0
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/ts_tablet_manager-itest.cc:768: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
  Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/ts_tablet_manager-itest.cc:786: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 786, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 786, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
  Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/ts_tablet_manager-itest.cc:896: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20251024 20:01:59.603470 19045 tablet_server.cc:178] TabletServer@127.18.153.65:0 shutting down...
I20251024 20:01:59.607218 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:59.607338 19045 tablet_replica.cc:333] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679: stopping tablet replica
I20251024 20:01:59.607427 19045 raft_consensus.cc:2243] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 2 LEADER]: Raft consensus shutting down.
I20251024 20:01:59.607492 19045 raft_consensus.cc:2272] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.607657 19045 tablet_replica.cc:333] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679: stopping tablet replica
I20251024 20:01:59.607687 19045 raft_consensus.cc:2243] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 3 LEADER]: Raft consensus shutting down.
I20251024 20:01:59.607717 19045 raft_consensus.cc:2272] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.607818 19045 tablet_replica.cc:333] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679: stopping tablet replica
I20251024 20:01:59.607847 19045 raft_consensus.cc:2243] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 3 LEADER]: Raft consensus shutting down.
I20251024 20:01:59.607882 19045 raft_consensus.cc:2272] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.619843 19045 tablet_server.cc:195] TabletServer@127.18.153.65:0 shutdown complete.
I20251024 20:01:59.620924 19045 tablet_server.cc:178] TabletServer@127.18.153.66:0 shutting down...
I20251024 20:01:59.624572 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:59.624722 19045 tablet_replica.cc:333] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182: stopping tablet replica
I20251024 20:01:59.624792 19045 raft_consensus.cc:2243] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.624856 19045 raft_consensus.cc:2272] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.625110 19045 tablet_replica.cc:333] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182: stopping tablet replica
I20251024 20:01:59.625154 19045 raft_consensus.cc:2243] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.625211 19045 raft_consensus.cc:2272] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.625335 19045 tablet_replica.cc:333] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182: stopping tablet replica
I20251024 20:01:59.625370 19045 raft_consensus.cc:2243] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.625409 19045 raft_consensus.cc:2272] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.637120 19045 tablet_server.cc:195] TabletServer@127.18.153.66:0 shutdown complete.
I20251024 20:01:59.638175 19045 tablet_server.cc:178] TabletServer@127.18.153.67:0 shutting down...
I20251024 20:01:59.641719 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:59.641884 19045 tablet_replica.cc:333] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4: stopping tablet replica
I20251024 20:01:59.641952 19045 raft_consensus.cc:2243] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.642010 19045 raft_consensus.cc:2272] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.642186 19045 tablet_replica.cc:333] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4: stopping tablet replica
I20251024 20:01:59.642225 19045 raft_consensus.cc:2243] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.642266 19045 raft_consensus.cc:2272] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.642395 19045 tablet_replica.cc:333] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4: stopping tablet replica
I20251024 20:01:59.642429 19045 raft_consensus.cc:2243] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.642470 19045 raft_consensus.cc:2272] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.654212 19045 tablet_server.cc:195] TabletServer@127.18.153.67:0 shutdown complete.
I20251024 20:01:59.655160 19045 master.cc:561] Master@127.18.153.126:38517 shutting down...
I20251024 20:01:59.658114 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 6 LEADER]: Raft consensus shutting down.
I20251024 20:01:59.658205 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 6 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.658229 19045 tablet_replica.cc:333] stopping tablet replica
I20251024 20:01:59.670795 19045 master.cc:583] Master@127.18.153.126:38517 shutdown complete.
I20251024 20:01:59.672053 19045 master.cc:561] Master@127.18.153.125:38773 shutting down...
I20251024 20:01:59.674906 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 6 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.674978 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 6 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.675016 19045 tablet_replica.cc:333] stopping tablet replica
I20251024 20:01:59.688220 19045 master.cc:583] Master@127.18.153.125:38773 shutdown complete.
I20251024 20:01:59.689450 19045 master.cc:561] Master@127.18.153.124:38423 shutting down...
I20251024 20:01:59.692299 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 6 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.692397 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 6 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.692437 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a: stopping tablet replica
I20251024 20:01:59.704779 19045 master.cc:583] Master@127.18.153.124:38423 shutdown complete.
I20251024 20:01:59.706573 19045 test_util.cc:183] -----------------------------------------------
I20251024 20:01:59.706645 19045 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0

[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN      ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251024 20:00:53.698000 19045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.153.126:37079
I20251024 20:00:53.698323 19045 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 20:00:53.698485 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:00:53.700907 19057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:00:53.700930 19055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:00:53.701017 19054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:00:53.701084 19045 server_base.cc:1047] running on GCE node
I20251024 20:00:53.701272 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:00:53.701310 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:00:53.701337 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336053701337 us; error 0 us; skew 500 ppm
I20251024 20:00:53.702312 19045 webserver.cc:492] Webserver started at http://127.18.153.126:42627/ using document root <none> and password file <none>
I20251024 20:00:53.702435 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:00:53.702474 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:00:53.702540 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:00:53.703632 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/master-0-root/instance:
uuid: "fe10c59c74ee4d4a9b306512f47cf74d"
format_stamp: "Formatted at 2025-10-24 20:00:53 on dist-test-slave-375r"
I20251024 20:00:53.704798 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:00:53.705327 19062 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:00:53.705493 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:00:53.705562 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/master-0-root
uuid: "fe10c59c74ee4d4a9b306512f47cf74d"
format_stamp: "Formatted at 2025-10-24 20:00:53 on dist-test-slave-375r"
I20251024 20:00:53.705611 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:00:53.726307 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:00:53.726568 19045 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 20:00:53.726661 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:00:53.730712 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.126:37079
I20251024 20:00:53.730813 19114 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.126:37079 every 8 connection(s)
I20251024 20:00:53.731317 19115 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:00:53.733741 19115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d: Bootstrap starting.
I20251024 20:00:53.734617 19115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d: Neither blocks nor log segments found. Creating new log.
I20251024 20:00:53.734875 19115 log.cc:826] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d: Log is configured to *not* fsync() on all Append() calls
I20251024 20:00:53.735538 19115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d: No bootstrap required, opened a new log
I20251024 20:00:53.736732 19115 raft_consensus.cc:359] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe10c59c74ee4d4a9b306512f47cf74d" member_type: VOTER }
I20251024 20:00:53.736835 19115 raft_consensus.cc:385] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:00:53.736867 19115 raft_consensus.cc:740] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe10c59c74ee4d4a9b306512f47cf74d, State: Initialized, Role: FOLLOWER
I20251024 20:00:53.736999 19115 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe10c59c74ee4d4a9b306512f47cf74d" member_type: VOTER }
I20251024 20:00:53.737078 19115 raft_consensus.cc:399] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:00:53.737121 19115 raft_consensus.cc:493] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:00:53.737170 19115 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:00:53.737679 19115 raft_consensus.cc:515] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe10c59c74ee4d4a9b306512f47cf74d" member_type: VOTER }
I20251024 20:00:53.737779 19115 leader_election.cc:304] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe10c59c74ee4d4a9b306512f47cf74d; no voters: 
I20251024 20:00:53.737938 19115 leader_election.cc:290] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:00:53.737986 19118 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:00:53.738145 19118 raft_consensus.cc:697] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [term 1 LEADER]: Becoming Leader. State: Replica: fe10c59c74ee4d4a9b306512f47cf74d, State: Running, Role: LEADER
I20251024 20:00:53.738181 19115 sys_catalog.cc:565] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:00:53.738225 19118 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe10c59c74ee4d4a9b306512f47cf74d" member_type: VOTER }
I20251024 20:00:53.738715 19120 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe10c59c74ee4d4a9b306512f47cf74d. Latest consensus state: current_term: 1 leader_uuid: "fe10c59c74ee4d4a9b306512f47cf74d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe10c59c74ee4d4a9b306512f47cf74d" member_type: VOTER } }
I20251024 20:00:53.738692 19119 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe10c59c74ee4d4a9b306512f47cf74d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe10c59c74ee4d4a9b306512f47cf74d" member_type: VOTER } }
I20251024 20:00:53.738821 19119 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [sys.catalog]: This master's current role is: LEADER
I20251024 20:00:53.738970 19120 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [sys.catalog]: This master's current role is: LEADER
I20251024 20:00:53.739216 19123 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:00:53.739710 19123 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:00:53.739866 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251024 20:00:53.741024 19123 catalog_manager.cc:1357] Generated new cluster ID: 7bd519d92eef4f7abc1c825b7791f917
I20251024 20:00:53.741067 19123 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:00:53.760640 19123 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 20:00:53.761212 19123 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:00:53.768617 19123 catalog_manager.cc:6022] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d: Generated new TSK 0
I20251024 20:00:53.768800 19123 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:00:53.771693 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:00:53.773200 19138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:00:53.773240 19137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:00:53.773308 19140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:00:53.773458 19045 server_base.cc:1047] running on GCE node
I20251024 20:00:53.773685 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:00:53.773715 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:00:53.773736 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336053773736 us; error 0 us; skew 500 ppm
I20251024 20:00:53.774370 19045 webserver.cc:492] Webserver started at http://127.18.153.65:42275/ using document root <none> and password file <none>
I20251024 20:00:53.774463 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:00:53.774513 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:00:53.774562 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:00:53.774823 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/ts-0-root/instance:
uuid: "d858b68b5a9f48d4a362b3a420f74aa1"
format_stamp: "Formatted at 2025-10-24 20:00:53 on dist-test-slave-375r"
I20251024 20:00:53.775755 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:00:53.776197 19145 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:00:53.776352 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:00:53.776417 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "d858b68b5a9f48d4a362b3a420f74aa1"
format_stamp: "Formatted at 2025-10-24 20:00:53 on dist-test-slave-375r"
I20251024 20:00:53.776463 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:00:53.792573 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:00:53.792907 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:00:53.793133 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:00:53.793454 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:00:53.793502 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:00:53.793538 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:00:53.793565 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:00:53.798019 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:45301
I20251024 20:00:53.798071 19208 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:45301 every 8 connection(s)
I20251024 20:00:53.798481 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:00:53.799932 19213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:00:53.800125 19212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:00:53.800187 19215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:00:53.800877 19045 server_base.cc:1047] running on GCE node
I20251024 20:00:53.800998 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:00:53.801030 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:00:53.801047 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336053801047 us; error 0 us; skew 500 ppm
I20251024 20:00:53.801770 19045 webserver.cc:492] Webserver started at http://127.18.153.66:33345/ using document root <none> and password file <none>
I20251024 20:00:53.801877 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:00:53.801928 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:00:53.801980 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:00:53.802330 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/ts-1-root/instance:
uuid: "455fd02c5a8846c38dc14d7ea4f995fe"
format_stamp: "Formatted at 2025-10-24 20:00:53 on dist-test-slave-375r"
I20251024 20:00:53.803331 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:00:53.803826 19221 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:00:53.803948 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:00:53.803997 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/ts-1-root
uuid: "455fd02c5a8846c38dc14d7ea4f995fe"
format_stamp: "Formatted at 2025-10-24 20:00:53 on dist-test-slave-375r"
I20251024 20:00:53.804056 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1761336053687453-19045-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:00:53.807852 19209 heartbeater.cc:344] Connected to a master server at 127.18.153.126:37079
I20251024 20:00:53.807960 19209 heartbeater.cc:461] Registering TS with master...
I20251024 20:00:53.808149 19209 heartbeater.cc:507] Master 127.18.153.126:37079 requested a full tablet report, sending...
I20251024 20:00:53.808601 19079 ts_manager.cc:194] Registered new tserver with Master: d858b68b5a9f48d4a362b3a420f74aa1 (127.18.153.65:45301)
I20251024 20:00:53.809623 19079 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50242
I20251024 20:00:53.824389 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:00:53.824636 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:00:53.824978 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:00:53.825202 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:00:53.825237 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:00:53.825273 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:00:53.825299 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:00:53.828596 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.66:42733
I20251024 20:00:53.828646 19284 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.66:42733 every 8 connection(s)
I20251024 20:00:53.832193 19285 heartbeater.cc:344] Connected to a master server at 127.18.153.126:37079
I20251024 20:00:53.832274 19285 heartbeater.cc:461] Registering TS with master...
I20251024 20:00:53.832399 19285 heartbeater.cc:507] Master 127.18.153.126:37079 requested a full tablet report, sending...
I20251024 20:00:53.832641 19079 ts_manager.cc:194] Registered new tserver with Master: 455fd02c5a8846c38dc14d7ea4f995fe (127.18.153.66:42733)
I20251024 20:00:53.832968 19079 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50256
I20251024 20:00:53.833235 19045 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.004431891s
I20251024 20:00:53.840466 19079 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50266:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 2
    seed: 0
  }
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20251024 20:00:53.841730 19079 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20251024 20:00:53.847884 19173 tablet_service.cc:1505] Processing CreateTablet for tablet 5ae5d4e52f76419cbba7fec2ddd2310a (DEFAULT_TABLE table=test-table [id=0af3a0f233bd437e8a0453dd8e4de6c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251024 20:00:53.848037 19249 tablet_service.cc:1505] Processing CreateTablet for tablet 5ae5d4e52f76419cbba7fec2ddd2310a (DEFAULT_TABLE table=test-table [id=0af3a0f233bd437e8a0453dd8e4de6c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251024 20:00:53.848117 19249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae5d4e52f76419cbba7fec2ddd2310a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:00:53.847884 19172 tablet_service.cc:1505] Processing CreateTablet for tablet d03ae12a5877486d90abee62288947c2 (DEFAULT_TABLE table=test-table [id=0af3a0f233bd437e8a0453dd8e4de6c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251024 20:00:53.848124 19173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae5d4e52f76419cbba7fec2ddd2310a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:00:53.849330 19248 tablet_service.cc:1505] Processing CreateTablet for tablet d03ae12a5877486d90abee62288947c2 (DEFAULT_TABLE table=test-table [id=0af3a0f233bd437e8a0453dd8e4de6c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251024 20:00:53.849656 19248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d03ae12a5877486d90abee62288947c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:00:53.850831 19172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d03ae12a5877486d90abee62288947c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:00:53.851166 19299 tablet_bootstrap.cc:492] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1: Bootstrap starting.
I20251024 20:00:53.851657 19299 tablet_bootstrap.cc:654] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1: Neither blocks nor log segments found. Creating new log.
I20251024 20:00:53.852265 19299 tablet_bootstrap.cc:492] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1: No bootstrap required, opened a new log
I20251024 20:00:53.852334 19299 ts_tablet_manager.cc:1403] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1: Time spent bootstrapping tablet: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:00:53.852708 19299 raft_consensus.cc:359] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:00:53.852756 19299 raft_consensus.cc:740] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d858b68b5a9f48d4a362b3a420f74aa1, State: Initialized, Role: FOLLOWER
I20251024 20:00:53.852777 19299 consensus_queue.cc:260] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:00:53.852916 19299 ts_tablet_manager.cc:1434] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20251024 20:00:53.853010 19299 tablet_bootstrap.cc:492] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1: Bootstrap starting.
I20251024 20:00:53.853021 19209 heartbeater.cc:499] Master 127.18.153.126:37079 was elected leader, sending a full tablet report...
I20251024 20:00:53.853431 19299 tablet_bootstrap.cc:654] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1: Neither blocks nor log segments found. Creating new log.
I20251024 20:00:53.854209 19299 tablet_bootstrap.cc:492] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1: No bootstrap required, opened a new log
I20251024 20:00:53.854257 19299 ts_tablet_manager.cc:1403] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1: Time spent bootstrapping tablet: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:00:53.854394 19299 raft_consensus.cc:359] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:00:53.854465 19299 raft_consensus.cc:740] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d858b68b5a9f48d4a362b3a420f74aa1, State: Initialized, Role: FOLLOWER
I20251024 20:00:53.854497 19299 consensus_queue.cc:260] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:00:53.854598 19299 ts_tablet_manager.cc:1434] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1: Time spent starting tablet: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:00:53.849335 19298 tablet_bootstrap.cc:492] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe: Bootstrap starting.
I20251024 20:00:53.855296 19298 tablet_bootstrap.cc:654] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe: Neither blocks nor log segments found. Creating new log.
I20251024 20:00:53.856072 19298 tablet_bootstrap.cc:492] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe: No bootstrap required, opened a new log
I20251024 20:00:53.856120 19298 ts_tablet_manager.cc:1403] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe: Time spent bootstrapping tablet: real 0.007s	user 0.001s	sys 0.000s
I20251024 20:00:53.856248 19298 raft_consensus.cc:359] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:00:53.856300 19298 raft_consensus.cc:740] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Initialized, Role: FOLLOWER
I20251024 20:00:53.856333 19298 consensus_queue.cc:260] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:00:53.856500 19298 ts_tablet_manager.cc:1434] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe: Time spent starting tablet: real 0.000s	user 0.001s	sys 0.000s
I20251024 20:00:53.856534 19285 heartbeater.cc:499] Master 127.18.153.126:37079 was elected leader, sending a full tablet report...
I20251024 20:00:53.856562 19298 tablet_bootstrap.cc:492] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe: Bootstrap starting.
I20251024 20:00:53.857026 19298 tablet_bootstrap.cc:654] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe: Neither blocks nor log segments found. Creating new log.
I20251024 20:00:53.857493 19298 tablet_bootstrap.cc:492] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe: No bootstrap required, opened a new log
I20251024 20:00:53.857539 19298 ts_tablet_manager.cc:1403] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:00:53.857667 19298 raft_consensus.cc:359] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:00:53.857703 19298 raft_consensus.cc:740] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Initialized, Role: FOLLOWER
I20251024 20:00:53.857721 19298 consensus_queue.cc:260] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:00:53.857795 19298 ts_tablet_manager.cc:1434] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe: Time spent starting tablet: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:00:53.864440 19045 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20251024 20:00:53.864500 19045 raft_consensus.cc:423] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 0 FOLLOWER]: Emulating election...
I20251024 20:00:53.864526 19045 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:00:53.865240 19045 raft_consensus.cc:697] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 1 LEADER]: Becoming Leader. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:00:53.865299 19045 consensus_queue.cc:237] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:00:53.865585 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:00:53.866164 19078 catalog_manager.cc:5649] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 455fd02c5a8846c38dc14d7ea4f995fe (127.18.153.66). New cstate: current_term: 1 leader_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: HEALTHY } } }
I20251024 20:00:53.871762 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20251024 20:00:53.972468 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
W20251024 20:00:54.048463 19210 tablet.cc:2378] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 20:00:54.173556 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20251024 20:00:54.423847 19183 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:00:54.424950 19309 consensus_queue.cc:1048] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:00:54.474761 19045 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20251024 20:00:54.474845 19045 raft_consensus.cc:423] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 0 FOLLOWER]: Emulating election...
I20251024 20:00:54.474869 19045 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:00:54.475549 19045 raft_consensus.cc:697] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 1 LEADER]: Becoming Leader. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:00:54.475616 19045 consensus_queue.cc:237] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:00:54.475888 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:00:54.476229 19079 catalog_manager.cc:5649] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 455fd02c5a8846c38dc14d7ea4f995fe (127.18.153.66). New cstate: current_term: 1 leader_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: UNKNOWN } } }
I20251024 20:00:54.476495 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20251024 20:00:54.577553 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20251024 20:00:54.778406 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20251024 20:00:55.077468 19183 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:00:55.078452 19309 consensus_queue.cc:1048] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:00:55.279592 19045 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20251024 20:00:55.279680 19045 raft_consensus.cc:423] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 1 LEADER]: Emulating election...
I20251024 20:00:55.279706 19045 raft_consensus.cc:3055] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 1 LEADER]: Stepping down as leader of term 1
I20251024 20:00:55.279718 19045 raft_consensus.cc:740] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:00:55.279760 19045 consensus_queue.cc:260] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:00:55.279897 19045 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:00:55.280534 19045 raft_consensus.cc:697] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 2 LEADER]: Becoming Leader. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:00:55.280594 19045 consensus_queue.cc:237] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:00:55.280791 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:00:55.281136 19079 catalog_manager.cc:5649] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: UNKNOWN } } }
I20251024 20:00:55.281571 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 1.1 2.2
I20251024 20:00:55.382310 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 1.1 2.2
I20251024 20:00:55.583416 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 1.1 2.2
I20251024 20:00:55.716362 19183 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:00:55.717108 19183 raft_consensus.cc:1275] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 2 FOLLOWER]: Refusing update from remote peer 455fd02c5a8846c38dc14d7ea4f995fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251024 20:00:55.717376 19323 consensus_queue.cc:1048] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251024 20:00:55.884548 19045 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:00:55.884621 19045 raft_consensus.cc:423] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 1 FOLLOWER]: Emulating election...
I20251024 20:00:55.884646 19045 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:00:55.885376 19045 raft_consensus.cc:697] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 2 LEADER]: Becoming Leader. State: Replica: d858b68b5a9f48d4a362b3a420f74aa1, State: Running, Role: LEADER
I20251024 20:00:55.885457 19045 consensus_queue.cc:237] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:00:55.885695 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:00:55.885960 19079 catalog_manager.cc:5649] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 reported cstate change: term changed from 1 to 2, leader changed from 455fd02c5a8846c38dc14d7ea4f995fe (127.18.153.66) to d858b68b5a9f48d4a362b3a420f74aa1 (127.18.153.65). New cstate: current_term: 2 leader_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: UNKNOWN } } }
I20251024 20:00:55.886345 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 1.1
I20251024 20:00:55.987614 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 1.1
I20251024 20:00:56.188598 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 1.1
I20251024 20:00:56.325343 19259 raft_consensus.cc:3055] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 1 LEADER]: Stepping down as leader of term 1
I20251024 20:00:56.325425 19259 raft_consensus.cc:740] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:00:56.325459 19259 consensus_queue.cc:260] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:00:56.325534 19259 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:00:56.326316 19259 raft_consensus.cc:1275] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 2 FOLLOWER]: Refusing update from remote peer d858b68b5a9f48d4a362b3a420f74aa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251024 20:00:56.326556 19329 consensus_queue.cc:1048] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251024 20:00:56.689833 19045 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20251024 20:00:56.689935 19045 raft_consensus.cc:423] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 2 FOLLOWER]: Emulating election...
I20251024 20:00:56.689980 19045 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:00:56.690629 19045 raft_consensus.cc:697] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 3 LEADER]: Becoming Leader. State: Replica: d858b68b5a9f48d4a362b3a420f74aa1, State: Running, Role: LEADER
I20251024 20:00:56.690716 19045 consensus_queue.cc:237] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:00:56.690908 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:00:56.691304 19079 catalog_manager.cc:5649] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 reported cstate change: term changed from 2 to 3, leader changed from 455fd02c5a8846c38dc14d7ea4f995fe (127.18.153.66) to d858b68b5a9f48d4a362b3a420f74aa1 (127.18.153.65). New cstate: current_term: 3 leader_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: HEALTHY } } }
I20251024 20:00:56.691553 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 2.2
I20251024 20:00:56.792706 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 2.2
I20251024 20:00:56.805233 19183 raft_consensus.cc:1240] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 3 LEADER]: Rejecting Update request from peer 455fd02c5a8846c38dc14d7ea4f995fe for earlier term 2. Current term is 3. Ops: []
I20251024 20:00:56.805501 19323 consensus_queue.cc:1059] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:00:56.805649 19323 consensus_queue.cc:1243] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Peer d858b68b5a9f48d4a362b3a420f74aa1 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251024 20:00:56.805686 19323 raft_consensus.cc:3055] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 2 LEADER]: Stepping down as leader of term 2
I20251024 20:00:56.805709 19323 raft_consensus.cc:740] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:00:56.805756 19323 consensus_queue.cc:260] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:00:56.805826 19323 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:00:56.993515 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 2.2
I20251024 20:00:57.252254 19259 raft_consensus.cc:1275] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 3 FOLLOWER]: Refusing update from remote peer d858b68b5a9f48d4a362b3a420f74aa1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251024 20:00:57.252534 19329 consensus_queue.cc:1048] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:00:57.294293 19045 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20251024 20:00:57.294368 19045 raft_consensus.cc:423] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 3 FOLLOWER]: Emulating election...
I20251024 20:00:57.294390 19045 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 3 FOLLOWER]: Advancing to term 4
I20251024 20:00:57.294983 19045 raft_consensus.cc:697] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 4 LEADER]: Becoming Leader. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:00:57.295064 19045 consensus_queue.cc:237] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:00:57.295251 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:00:57.295524 19078 catalog_manager.cc:5649] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe reported cstate change: term changed from 3 to 4, leader changed from d858b68b5a9f48d4a362b3a420f74aa1 (127.18.153.65) to 455fd02c5a8846c38dc14d7ea4f995fe (127.18.153.66). New cstate: current_term: 4 leader_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: UNKNOWN } } }
I20251024 20:00:57.295850 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20251024 20:00:57.396755 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20251024 20:00:57.598208 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20251024 20:00:57.718693 19259 raft_consensus.cc:1240] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 4 LEADER]: Rejecting Update request from peer d858b68b5a9f48d4a362b3a420f74aa1 for earlier term 3. Current term is 4. Ops: []
I20251024 20:00:57.718979 19329 consensus_queue.cc:1059] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251024 20:00:57.719101 19329 consensus_queue.cc:1243] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Peer 455fd02c5a8846c38dc14d7ea4f995fe log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20251024 20:00:57.719125 19329 raft_consensus.cc:3055] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 3 LEADER]: Stepping down as leader of term 3
I20251024 20:00:57.719158 19329 raft_consensus.cc:740] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d858b68b5a9f48d4a362b3a420f74aa1, State: Running, Role: LEADER
I20251024 20:00:57.719202 19329 consensus_queue.cc:260] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:00:57.719272 19329 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 3 FOLLOWER]: Advancing to term 4
I20251024 20:00:57.766877 19183 raft_consensus.cc:1275] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 4 FOLLOWER]: Refusing update from remote peer 455fd02c5a8846c38dc14d7ea4f995fe: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251024 20:00:57.767150 19323 consensus_queue.cc:1048] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251024 20:00:58.099233 19045 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20251024 20:00:58.099324 19045 raft_consensus.cc:423] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 4 LEADER]: Emulating election...
I20251024 20:00:58.099354 19045 raft_consensus.cc:3055] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 4 LEADER]: Stepping down as leader of term 4
I20251024 20:00:58.099409 19045 raft_consensus.cc:740] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:00:58.099464 19045 consensus_queue.cc:260] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:00:58.099566 19045 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 4 FOLLOWER]: Advancing to term 5
I20251024 20:00:58.100293 19045 raft_consensus.cc:697] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 5 LEADER]: Becoming Leader. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:00:58.100361 19045 consensus_queue.cc:237] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:00:58.100509 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:00:58.100759 19079 catalog_manager.cc:5649] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe reported cstate change: term changed from 4 to 5. New cstate: current_term: 5 leader_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: UNKNOWN } } }
I20251024 20:00:58.101220 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 4.4 5.5
I20251024 20:00:58.202258 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 4.4 5.5
I20251024 20:00:58.403020 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 4.4 5.5
I20251024 20:00:58.632488 19183 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 4 FOLLOWER]: Advancing to term 5
I20251024 20:00:58.633211 19183 raft_consensus.cc:1275] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 5 FOLLOWER]: Refusing update from remote peer 455fd02c5a8846c38dc14d7ea4f995fe: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251024 20:00:58.633466 19361 consensus_queue.cc:1048] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251024 20:00:58.704093 19045 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20251024 20:00:58.704159 19045 raft_consensus.cc:423] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 2 FOLLOWER]: Emulating election...
I20251024 20:00:58.704181 19045 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:00:58.704790 19045 raft_consensus.cc:697] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 3 LEADER]: Becoming Leader. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:00:58.704869 19045 consensus_queue.cc:237] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:00:58.705221 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:00:58.705528 19078 catalog_manager.cc:5649] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe reported cstate change: term changed from 2 to 3, leader changed from d858b68b5a9f48d4a362b3a420f74aa1 (127.18.153.65) to 455fd02c5a8846c38dc14d7ea4f995fe (127.18.153.66). New cstate: current_term: 3 leader_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: HEALTHY } } }
I20251024 20:00:58.705755 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 3.3
I20251024 20:00:58.806578 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 3.3
I20251024 20:00:58.956385 19259 raft_consensus.cc:1240] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 3 LEADER]: Rejecting Update request from peer d858b68b5a9f48d4a362b3a420f74aa1 for earlier term 2. Current term is 3. Ops: []
I20251024 20:00:58.956661 19360 consensus_queue.cc:1059] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:00:58.956903 19360 consensus_queue.cc:1243] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Peer 455fd02c5a8846c38dc14d7ea4f995fe log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251024 20:00:58.956957 19369 raft_consensus.cc:3055] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 2 LEADER]: Stepping down as leader of term 2
I20251024 20:00:58.957000 19369 raft_consensus.cc:740] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d858b68b5a9f48d4a362b3a420f74aa1, State: Running, Role: LEADER
I20251024 20:00:58.957039 19369 consensus_queue.cc:260] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:00:58.957110 19369 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:00:59.007414 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 3.3
I20251024 20:00:59.091451 19183 raft_consensus.cc:1275] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 3 FOLLOWER]: Refusing update from remote peer 455fd02c5a8846c38dc14d7ea4f995fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251024 20:00:59.091732 19361 consensus_queue.cc:1048] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:00:59.508260 19045 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20251024 20:00:59.508338 19045 raft_consensus.cc:423] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 3 LEADER]: Emulating election...
I20251024 20:00:59.508364 19045 raft_consensus.cc:3055] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 3 LEADER]: Stepping down as leader of term 3
I20251024 20:00:59.508376 19045 raft_consensus.cc:740] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:00:59.508417 19045 consensus_queue.cc:260] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:00:59.508500 19045 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 3 FOLLOWER]: Advancing to term 4
I20251024 20:00:59.509066 19045 raft_consensus.cc:697] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 4 LEADER]: Becoming Leader. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:00:59.509135 19045 consensus_queue.cc:237] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:00:59.509285 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:00:59.509601 19078 catalog_manager.cc:5649] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe reported cstate change: term changed from 3 to 4. New cstate: current_term: 4 leader_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: HEALTHY } } }
I20251024 20:00:59.509958 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20251024 20:00:59.610883 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20251024 20:00:59.811723 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20251024 20:00:59.983409 19183 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 3 FOLLOWER]: Advancing to term 4
I20251024 20:00:59.984118 19183 raft_consensus.cc:1275] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 4 FOLLOWER]: Refusing update from remote peer 455fd02c5a8846c38dc14d7ea4f995fe: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251024 20:00:59.984382 19361 consensus_queue.cc:1048] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251024 20:01:00.112550 19045 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:00.112623 19045 raft_consensus.cc:423] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 4 FOLLOWER]: Emulating election...
I20251024 20:01:00.112649 19045 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 4 FOLLOWER]: Advancing to term 5
I20251024 20:01:00.114056 19045 raft_consensus.cc:697] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 5 LEADER]: Becoming Leader. State: Replica: d858b68b5a9f48d4a362b3a420f74aa1, State: Running, Role: LEADER
I20251024 20:01:00.114137 19045 consensus_queue.cc:237] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:01:00.114303 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:00.114625 19079 catalog_manager.cc:5649] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 reported cstate change: term changed from 4 to 5, leader changed from 455fd02c5a8846c38dc14d7ea4f995fe (127.18.153.66) to d858b68b5a9f48d4a362b3a420f74aa1 (127.18.153.65). New cstate: current_term: 5 leader_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: UNKNOWN } } }
I20251024 20:01:00.114857 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 5.5 4.4
I20251024 20:01:00.216162 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 5.5 4.4
I20251024 20:01:00.417371 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 5.5 4.4
I20251024 20:01:00.471812 19183 raft_consensus.cc:1240] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 5 LEADER]: Rejecting Update request from peer 455fd02c5a8846c38dc14d7ea4f995fe for earlier term 4. Current term is 5. Ops: []
I20251024 20:01:00.472121 19375 consensus_queue.cc:1059] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251024 20:01:00.472234 19375 consensus_queue.cc:1243] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Peer d858b68b5a9f48d4a362b3a420f74aa1 log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251024 20:01:00.472268 19375 raft_consensus.cc:3055] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 4 LEADER]: Stepping down as leader of term 4
I20251024 20:01:00.472299 19375 raft_consensus.cc:740] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:01:00.472337 19375 consensus_queue.cc:260] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:01:00.472389 19375 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 4 FOLLOWER]: Advancing to term 5
I20251024 20:01:00.618755 19259 raft_consensus.cc:1275] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 5 FOLLOWER]: Refusing update from remote peer d858b68b5a9f48d4a362b3a420f74aa1: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251024 20:01:00.619006 19381 consensus_queue.cc:1048] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251024 20:01:00.918439 19045 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:00.918519 19045 raft_consensus.cc:423] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 5 LEADER]: Emulating election...
I20251024 20:01:00.918555 19045 raft_consensus.cc:3055] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 5 LEADER]: Stepping down as leader of term 5
I20251024 20:01:00.918571 19045 raft_consensus.cc:740] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d858b68b5a9f48d4a362b3a420f74aa1, State: Running, Role: LEADER
I20251024 20:01:00.918617 19045 consensus_queue.cc:260] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:01:00.918713 19045 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 5 FOLLOWER]: Advancing to term 6
I20251024 20:01:00.919314 19045 raft_consensus.cc:697] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 6 LEADER]: Becoming Leader. State: Replica: d858b68b5a9f48d4a362b3a420f74aa1, State: Running, Role: LEADER
I20251024 20:01:00.919410 19045 consensus_queue.cc:237] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:01:00.919585 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:00.919878 19078 catalog_manager.cc:5649] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 reported cstate change: term changed from 5 to 6. New cstate: current_term: 6 leader_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: UNKNOWN } } }
I20251024 20:01:00.920228 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 6.6 5.5
I20251024 20:01:01.021394 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 6.6 5.5
I20251024 20:01:01.222343 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 6.6 5.5
I20251024 20:01:01.490566 19259 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 5 FOLLOWER]: Advancing to term 6
I20251024 20:01:01.491353 19259 raft_consensus.cc:1275] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 6 FOLLOWER]: Refusing update from remote peer d858b68b5a9f48d4a362b3a420f74aa1: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20251024 20:01:01.491647 19389 consensus_queue.cc:1048] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251024 20:01:01.523404 19045 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20251024 20:01:01.523494 19045 raft_consensus.cc:423] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 5 FOLLOWER]: Emulating election...
I20251024 20:01:01.523516 19045 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 5 FOLLOWER]: Advancing to term 6
I20251024 20:01:01.524138 19045 raft_consensus.cc:697] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 6 LEADER]: Becoming Leader. State: Replica: d858b68b5a9f48d4a362b3a420f74aa1, State: Running, Role: LEADER
I20251024 20:01:01.524204 19045 consensus_queue.cc:237] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:01:01.524444 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:01.524732 19078 catalog_manager.cc:5649] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 reported cstate change: term changed from 5 to 6, leader changed from 455fd02c5a8846c38dc14d7ea4f995fe (127.18.153.66) to d858b68b5a9f48d4a362b3a420f74aa1 (127.18.153.65). New cstate: current_term: 6 leader_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: HEALTHY } } }
I20251024 20:01:01.525049 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 6.6 5.5
I20251024 20:01:01.626223 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 6.6 5.5
I20251024 20:01:01.827507 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 6.6 5.5
I20251024 20:01:01.871264 19183 raft_consensus.cc:1240] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 6 LEADER]: Rejecting Update request from peer 455fd02c5a8846c38dc14d7ea4f995fe for earlier term 5. Current term is 6. Ops: []
I20251024 20:01:01.871603 19388 consensus_queue.cc:1059] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251024 20:01:01.871821 19388 consensus_queue.cc:1243] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Peer d858b68b5a9f48d4a362b3a420f74aa1 log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20251024 20:01:01.871861 19388 raft_consensus.cc:3055] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 5 LEADER]: Stepping down as leader of term 5
I20251024 20:01:01.871886 19388 raft_consensus.cc:740] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:01:01.871935 19388 consensus_queue.cc:260] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:01:01.872013 19388 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 5 FOLLOWER]: Advancing to term 6
I20251024 20:01:01.992281 19259 raft_consensus.cc:1275] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 6 FOLLOWER]: Refusing update from remote peer d858b68b5a9f48d4a362b3a420f74aa1: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20251024 20:01:01.992612 19389 consensus_queue.cc:1048] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251024 20:01:02.328873 19045 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20251024 20:01:02.328950 19045 raft_consensus.cc:423] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 6 FOLLOWER]: Emulating election...
I20251024 20:01:02.328974 19045 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 6 FOLLOWER]: Advancing to term 7
I20251024 20:01:02.329557 19045 raft_consensus.cc:697] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 7 LEADER]: Becoming Leader. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:01:02.329639 19045 consensus_queue.cc:237] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:01:02.329815 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:02.330097 19078 catalog_manager.cc:5649] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe reported cstate change: term changed from 6 to 7, leader changed from d858b68b5a9f48d4a362b3a420f74aa1 (127.18.153.65) to 455fd02c5a8846c38dc14d7ea4f995fe (127.18.153.66). New cstate: current_term: 7 leader_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: HEALTHY } } }
I20251024 20:01:02.330430 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 6.6 7.7
I20251024 20:01:02.431428 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 6.6 7.7
I20251024 20:01:02.515524 19259 raft_consensus.cc:1240] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 7 LEADER]: Rejecting Update request from peer d858b68b5a9f48d4a362b3a420f74aa1 for earlier term 6. Current term is 7. Ops: []
I20251024 20:01:02.515807 19399 consensus_queue.cc:1059] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251024 20:01:02.516000 19399 consensus_queue.cc:1243] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Peer 455fd02c5a8846c38dc14d7ea4f995fe log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20251024 20:01:02.516052 19405 raft_consensus.cc:3055] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 6 LEADER]: Stepping down as leader of term 6
I20251024 20:01:02.516114 19405 raft_consensus.cc:740] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d858b68b5a9f48d4a362b3a420f74aa1, State: Running, Role: LEADER
I20251024 20:01:02.516177 19405 consensus_queue.cc:260] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:01:02.516259 19405 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 6 FOLLOWER]: Advancing to term 7
I20251024 20:01:02.632274 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 6.6 7.7
I20251024 20:01:02.726006 19183 raft_consensus.cc:1275] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 7 FOLLOWER]: Refusing update from remote peer 455fd02c5a8846c38dc14d7ea4f995fe: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20251024 20:01:02.726300 19388 consensus_queue.cc:1048] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251024 20:01:02.933076 19045 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20251024 20:01:02.933156 19045 raft_consensus.cc:423] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 6 FOLLOWER]: Emulating election...
I20251024 20:01:02.933187 19045 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 6 FOLLOWER]: Advancing to term 7
I20251024 20:01:02.933784 19045 raft_consensus.cc:697] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 7 LEADER]: Becoming Leader. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:01:02.933848 19045 consensus_queue.cc:237] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:01:02.934019 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:02.934360 19079 catalog_manager.cc:5649] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe reported cstate change: term changed from 6 to 7, leader changed from d858b68b5a9f48d4a362b3a420f74aa1 (127.18.153.65) to 455fd02c5a8846c38dc14d7ea4f995fe (127.18.153.66). New cstate: current_term: 7 leader_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: UNKNOWN } } }
I20251024 20:01:02.934592 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 6.6 7.7
I20251024 20:01:03.035526 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 6.6 7.7
I20251024 20:01:03.236289 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 6.6 7.7
I20251024 20:01:03.333750 19259 raft_consensus.cc:1240] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 7 LEADER]: Rejecting Update request from peer d858b68b5a9f48d4a362b3a420f74aa1 for earlier term 6. Current term is 7. Ops: []
I20251024 20:01:03.334031 19405 consensus_queue.cc:1059] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251024 20:01:03.334233 19405 consensus_queue.cc:1243] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Peer 455fd02c5a8846c38dc14d7ea4f995fe log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20251024 20:01:03.334302 19413 raft_consensus.cc:3055] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 6 LEADER]: Stepping down as leader of term 6
I20251024 20:01:03.334362 19413 raft_consensus.cc:740] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d858b68b5a9f48d4a362b3a420f74aa1, State: Running, Role: LEADER
I20251024 20:01:03.334422 19413 consensus_queue.cc:260] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:01:03.334503 19413 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 6 FOLLOWER]: Advancing to term 7
I20251024 20:01:03.513105 19183 raft_consensus.cc:1275] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 7 FOLLOWER]: Refusing update from remote peer 455fd02c5a8846c38dc14d7ea4f995fe: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20251024 20:01:03.513396 19407 consensus_queue.cc:1048] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251024 20:01:03.737238 19045 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20251024 20:01:03.737335 19045 raft_consensus.cc:423] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 7 LEADER]: Emulating election...
I20251024 20:01:03.737365 19045 raft_consensus.cc:3055] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 7 LEADER]: Stepping down as leader of term 7
I20251024 20:01:03.737378 19045 raft_consensus.cc:740] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:01:03.737418 19045 consensus_queue.cc:260] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:01:03.737504 19045 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 7 FOLLOWER]: Advancing to term 8
I20251024 20:01:03.738087 19045 raft_consensus.cc:697] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 8 LEADER]: Becoming Leader. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:01:03.738162 19045 consensus_queue.cc:237] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:01:03.738320 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:03.738651 19078 catalog_manager.cc:5649] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe reported cstate change: term changed from 7 to 8. New cstate: current_term: 8 leader_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: UNKNOWN } } }
I20251024 20:01:03.738976 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 7.7 8.8
I20251024 20:01:03.839928 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 7.7 8.8
I20251024 20:01:04.040683 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 7.7 8.8
I20251024 20:01:04.216826 19183 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 7 FOLLOWER]: Advancing to term 8
I20251024 20:01:04.217593 19183 raft_consensus.cc:1275] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 8 FOLLOWER]: Refusing update from remote peer 455fd02c5a8846c38dc14d7ea4f995fe: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20251024 20:01:04.217837 19415 consensus_queue.cc:1048] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251024 20:01:04.341682 19045 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20251024 20:01:04.341758 19045 raft_consensus.cc:423] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 7 LEADER]: Emulating election...
I20251024 20:01:04.341780 19045 raft_consensus.cc:3055] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 7 LEADER]: Stepping down as leader of term 7
I20251024 20:01:04.341794 19045 raft_consensus.cc:740] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:01:04.341821 19045 consensus_queue.cc:260] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:01:04.341895 19045 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 7 FOLLOWER]: Advancing to term 8
I20251024 20:01:04.342500 19045 raft_consensus.cc:697] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 8 LEADER]: Becoming Leader. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:01:04.342561 19045 consensus_queue.cc:237] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:01:04.342784 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:04.343078 19079 catalog_manager.cc:5649] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe reported cstate change: term changed from 7 to 8. New cstate: current_term: 8 leader_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: HEALTHY } } }
I20251024 20:01:04.343406 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 7.7 8.8
I20251024 20:01:04.444133 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 7.7 8.8
I20251024 20:01:04.644922 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 7.7 8.8
I20251024 20:01:04.897843 19183 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 7 FOLLOWER]: Advancing to term 8
I20251024 20:01:04.898614 19183 raft_consensus.cc:1275] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 8 FOLLOWER]: Refusing update from remote peer 455fd02c5a8846c38dc14d7ea4f995fe: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20251024 20:01:04.898890 19422 consensus_queue.cc:1048] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251024 20:01:05.145934 19045 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20251024 20:01:05.146049 19045 raft_consensus.cc:423] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 8 LEADER]: Emulating election...
I20251024 20:01:05.146100 19045 raft_consensus.cc:3055] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 8 LEADER]: Stepping down as leader of term 8
I20251024 20:01:05.146126 19045 raft_consensus.cc:740] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:01:05.146195 19045 consensus_queue.cc:260] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:01:05.146293 19045 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 8 FOLLOWER]: Advancing to term 9
I20251024 20:01:05.148029 19045 raft_consensus.cc:697] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 9 LEADER]: Becoming Leader. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:01:05.148110 19045 consensus_queue.cc:237] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:01:05.148283 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:05.148728 19078 catalog_manager.cc:5649] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe reported cstate change: term changed from 8 to 9. New cstate: current_term: 9 leader_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: HEALTHY } } }
I20251024 20:01:05.148895 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 8.8 9.9
I20251024 20:01:05.249895 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 8.8 9.9
I20251024 20:01:05.450896 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 8.8 9.9
I20251024 20:01:05.650520 19183 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 8 FOLLOWER]: Advancing to term 9
I20251024 20:01:05.651264 19183 raft_consensus.cc:1275] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 9 FOLLOWER]: Refusing update from remote peer 455fd02c5a8846c38dc14d7ea4f995fe: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251024 20:01:05.651527 19431 consensus_queue.cc:1048] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251024 20:01:05.751886 19045 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20251024 20:01:05.751966 19045 raft_consensus.cc:423] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 8 LEADER]: Emulating election...
I20251024 20:01:05.751995 19045 raft_consensus.cc:3055] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 8 LEADER]: Stepping down as leader of term 8
I20251024 20:01:05.752015 19045 raft_consensus.cc:740] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:01:05.752065 19045 consensus_queue.cc:260] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:01:05.752161 19045 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 8 FOLLOWER]: Advancing to term 9
I20251024 20:01:05.752758 19045 raft_consensus.cc:697] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 9 LEADER]: Becoming Leader. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:01:05.752812 19045 consensus_queue.cc:237] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } }
I20251024 20:01:05.753062 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:05.753376 19079 catalog_manager.cc:5649] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe reported cstate change: term changed from 8 to 9. New cstate: current_term: 9 leader_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: UNKNOWN } } }
I20251024 20:01:05.753558 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 8.8 9.9
I20251024 20:01:05.854602 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 8.8 9.9
I20251024 20:01:06.055694 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 8.8 9.9
I20251024 20:01:06.261976 19183 raft_consensus.cc:3060] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 8 FOLLOWER]: Advancing to term 9
I20251024 20:01:06.262732 19183 raft_consensus.cc:1275] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 9 FOLLOWER]: Refusing update from remote peer 455fd02c5a8846c38dc14d7ea4f995fe: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251024 20:01:06.262956 19438 consensus_queue.cc:1048] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251024 20:01:06.556810 19045 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:06.556890 19045 raft_consensus.cc:423] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 9 FOLLOWER]: Emulating election...
I20251024 20:01:06.556913 19045 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 9 FOLLOWER]: Advancing to term 10
I20251024 20:01:06.557555 19045 raft_consensus.cc:697] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 10 LEADER]: Becoming Leader. State: Replica: d858b68b5a9f48d4a362b3a420f74aa1, State: Running, Role: LEADER
I20251024 20:01:06.557641 19045 consensus_queue.cc:237] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:01:06.557812 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:06.558085 19078 catalog_manager.cc:5649] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 reported cstate change: term changed from 9 to 10, leader changed from 455fd02c5a8846c38dc14d7ea4f995fe (127.18.153.66) to d858b68b5a9f48d4a362b3a420f74aa1 (127.18.153.65). New cstate: current_term: 10 leader_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: UNKNOWN } } }
I20251024 20:01:06.558528 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 10.10 9.9
I20251024 20:01:06.659543 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 10.10 9.9
I20251024 20:01:06.720381 19183 raft_consensus.cc:1240] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 10 LEADER]: Rejecting Update request from peer 455fd02c5a8846c38dc14d7ea4f995fe for earlier term 9. Current term is 10. Ops: []
I20251024 20:01:06.720644 19447 consensus_queue.cc:1059] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20251024 20:01:06.720760 19447 consensus_queue.cc:1243] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Peer d858b68b5a9f48d4a362b3a420f74aa1 log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20251024 20:01:06.720806 19438 raft_consensus.cc:3055] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 9 LEADER]: Stepping down as leader of term 9
I20251024 20:01:06.720890 19438 raft_consensus.cc:740] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:01:06.720940 19438 consensus_queue.cc:260] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:01:06.721024 19438 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 9 FOLLOWER]: Advancing to term 10
I20251024 20:01:06.860631 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 10.10 9.9
I20251024 20:01:07.040059 19259 raft_consensus.cc:1275] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 10 FOLLOWER]: Refusing update from remote peer d858b68b5a9f48d4a362b3a420f74aa1: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20251024 20:01:07.040326 19445 consensus_queue.cc:1048] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20251024 20:01:07.161481 19045 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20251024 20:01:07.161550 19045 raft_consensus.cc:423] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 10 FOLLOWER]: Emulating election...
I20251024 20:01:07.161573 19045 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 10 FOLLOWER]: Advancing to term 11
I20251024 20:01:07.162186 19045 raft_consensus.cc:697] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 11 LEADER]: Becoming Leader. State: Replica: 455fd02c5a8846c38dc14d7ea4f995fe, State: Running, Role: LEADER
I20251024 20:01:07.162254 19045 consensus_queue.cc:237] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:01:07.162417 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:07.162686 19078 catalog_manager.cc:5649] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe reported cstate change: term changed from 10 to 11, leader changed from d858b68b5a9f48d4a362b3a420f74aa1 (127.18.153.65) to 455fd02c5a8846c38dc14d7ea4f995fe (127.18.153.66). New cstate: current_term: 11 leader_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } health_report { overall_health: HEALTHY } } }
I20251024 20:01:07.162935 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 10.10 11.11
I20251024 20:01:07.264012 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 10.10 11.11
I20251024 20:01:07.465233 19045 cluster_itest_util.cc:257] Not converged past 0 yet: 10.10 11.11
I20251024 20:01:07.567939 19183 raft_consensus.cc:3055] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 10 LEADER]: Stepping down as leader of term 10
I20251024 20:01:07.568022 19183 raft_consensus.cc:740] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d858b68b5a9f48d4a362b3a420f74aa1, State: Running, Role: LEADER
I20251024 20:01:07.568051 19183 consensus_queue.cc:260] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 } } peers { permanent_uuid: "455fd02c5a8846c38dc14d7ea4f995fe" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42733 } }
I20251024 20:01:07.568132 19183 raft_consensus.cc:3060] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 10 FOLLOWER]: Advancing to term 11
I20251024 20:01:07.568748 19183 raft_consensus.cc:1275] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 11 FOLLOWER]: Refusing update from remote peer 455fd02c5a8846c38dc14d7ea4f995fe: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20251024 20:01:07.568994 19438 consensus_queue.cc:1048] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d858b68b5a9f48d4a362b3a420f74aa1" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20251024 20:01:07.966917 19045 tablet_server.cc:178] TabletServer@127.18.153.65:0 shutting down...
I20251024 20:01:07.970153 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:07.970316 19045 tablet_replica.cc:333] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1: stopping tablet replica
I20251024 20:01:07.970412 19045 raft_consensus.cc:2243] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 11 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:07.970463 19045 raft_consensus.cc:2272] T d03ae12a5877486d90abee62288947c2 P d858b68b5a9f48d4a362b3a420f74aa1 [term 11 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:07.970784 19045 tablet_replica.cc:333] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1: stopping tablet replica
I20251024 20:01:07.970824 19045 raft_consensus.cc:2243] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 9 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:07.970856 19045 raft_consensus.cc:2272] T 5ae5d4e52f76419cbba7fec2ddd2310a P d858b68b5a9f48d4a362b3a420f74aa1 [term 9 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:07.982928 19045 tablet_server.cc:195] TabletServer@127.18.153.65:0 shutdown complete.
I20251024 20:01:07.984033 19045 tablet_server.cc:178] TabletServer@127.18.153.66:0 shutting down...
I20251024 20:01:07.986598 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:07.986732 19045 tablet_replica.cc:333] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe: stopping tablet replica
I20251024 20:01:07.986806 19045 raft_consensus.cc:2243] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 11 LEADER]: Raft consensus shutting down.
I20251024 20:01:07.986862 19045 raft_consensus.cc:2272] T d03ae12a5877486d90abee62288947c2 P 455fd02c5a8846c38dc14d7ea4f995fe [term 11 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:07.987087 19045 tablet_replica.cc:333] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe: stopping tablet replica
I20251024 20:01:07.987125 19045 raft_consensus.cc:2243] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 9 LEADER]: Raft consensus shutting down.
I20251024 20:01:07.987159 19045 raft_consensus.cc:2272] T 5ae5d4e52f76419cbba7fec2ddd2310a P 455fd02c5a8846c38dc14d7ea4f995fe [term 9 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:07.998903 19045 tablet_server.cc:195] TabletServer@127.18.153.66:0 shutdown complete.
I20251024 20:01:07.999856 19045 master.cc:561] Master@127.18.153.126:37079 shutting down...
I20251024 20:01:08.002525 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:01:08.002607 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:08.002646 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P fe10c59c74ee4d4a9b306512f47cf74d: stopping tablet replica
I20251024 20:01:08.014621 19045 master.cc:583] Master@127.18.153.126:37079 shutdown complete.
[       OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (14326 ms)
[ RUN      ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20251024 20:01:08.019109 19045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.153.126:35939
I20251024 20:01:08.019275 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:08.020403 19468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:08.020473 19469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:08.020484 19471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:08.020709 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:08.020787 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:08.020817 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:08.020835 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336068020836 us; error 0 us; skew 500 ppm
I20251024 20:01:08.021413 19045 webserver.cc:492] Webserver started at http://127.18.153.126:40071/ using document root <none> and password file <none>
I20251024 20:01:08.021487 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:08.021529 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:08.021577 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:08.021842 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/master-0-root/instance:
uuid: "e2c7f49d524a4d5badb90f6ebc2aa5c3"
format_stamp: "Formatted at 2025-10-24 20:01:08 on dist-test-slave-375r"
I20251024 20:01:08.022707 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:08.023116 19476 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:08.023262 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:08.023312 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/master-0-root
uuid: "e2c7f49d524a4d5badb90f6ebc2aa5c3"
format_stamp: "Formatted at 2025-10-24 20:01:08 on dist-test-slave-375r"
I20251024 20:01:08.023357 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:08.029223 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:08.029435 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:08.032025 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.126:35939
I20251024 20:01:08.034004 19528 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.126:35939 every 8 connection(s)
I20251024 20:01:08.034169 19529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:08.035132 19529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3: Bootstrap starting.
I20251024 20:01:08.035418 19529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:08.035887 19529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3: No bootstrap required, opened a new log
I20251024 20:01:08.036007 19529 raft_consensus.cc:359] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c7f49d524a4d5badb90f6ebc2aa5c3" member_type: VOTER }
I20251024 20:01:08.036072 19529 raft_consensus.cc:385] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:08.036085 19529 raft_consensus.cc:740] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c7f49d524a4d5badb90f6ebc2aa5c3, State: Initialized, Role: FOLLOWER
I20251024 20:01:08.036129 19529 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c7f49d524a4d5badb90f6ebc2aa5c3" member_type: VOTER }
I20251024 20:01:08.036172 19529 raft_consensus.cc:399] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:01:08.036194 19529 raft_consensus.cc:493] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:01:08.036216 19529 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:08.036669 19529 raft_consensus.cc:515] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c7f49d524a4d5badb90f6ebc2aa5c3" member_type: VOTER }
I20251024 20:01:08.036724 19529 leader_election.cc:304] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2c7f49d524a4d5badb90f6ebc2aa5c3; no voters: 
I20251024 20:01:08.036810 19529 leader_election.cc:290] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:01:08.036876 19532 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:01:08.036939 19529 sys_catalog.cc:565] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:01:08.037041 19532 raft_consensus.cc:697] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [term 1 LEADER]: Becoming Leader. State: Replica: e2c7f49d524a4d5badb90f6ebc2aa5c3, State: Running, Role: LEADER
I20251024 20:01:08.037101 19532 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c7f49d524a4d5badb90f6ebc2aa5c3" member_type: VOTER }
I20251024 20:01:08.037319 19533 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2c7f49d524a4d5badb90f6ebc2aa5c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c7f49d524a4d5badb90f6ebc2aa5c3" member_type: VOTER } }
I20251024 20:01:08.037343 19534 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2c7f49d524a4d5badb90f6ebc2aa5c3. Latest consensus state: current_term: 1 leader_uuid: "e2c7f49d524a4d5badb90f6ebc2aa5c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c7f49d524a4d5badb90f6ebc2aa5c3" member_type: VOTER } }
I20251024 20:01:08.037393 19533 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:08.037401 19534 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:08.037750 19538 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:01:08.037916 19538 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:01:08.038290 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251024 20:01:08.038501 19538 catalog_manager.cc:1357] Generated new cluster ID: 67305511d5414dd09d599afcc6404618
I20251024 20:01:08.038545 19538 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:01:08.056978 19538 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 20:01:08.057427 19538 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:01:08.062034 19538 catalog_manager.cc:6022] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3: Generated new TSK 0
I20251024 20:01:08.062136 19538 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:01:08.070451 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:08.071985 19553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:08.072120 19550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:08.072275 19551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:08.072465 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:08.072569 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:08.072609 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:08.072640 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336068072640 us; error 0 us; skew 500 ppm
I20251024 20:01:08.073303 19045 webserver.cc:492] Webserver started at http://127.18.153.65:36463/ using document root <none> and password file <none>
I20251024 20:01:08.073398 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:08.073457 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:08.073519 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:08.073846 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/ts-0-root/instance:
uuid: "37d31b4fc9a947a0839f629659431c01"
format_stamp: "Formatted at 2025-10-24 20:01:08 on dist-test-slave-375r"
I20251024 20:01:08.074777 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:08.075284 19558 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:08.075429 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:08.075480 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "37d31b4fc9a947a0839f629659431c01"
format_stamp: "Formatted at 2025-10-24 20:01:08 on dist-test-slave-375r"
I20251024 20:01:08.075537 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:08.085017 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:08.085264 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:08.085434 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:08.085649 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:08.085683 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:08.085722 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:08.085753 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:08.088881 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:38815
I20251024 20:01:08.088923 19621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:38815 every 8 connection(s)
I20251024 20:01:08.089318 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:08.090615 19629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:08.090586 19626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:08.090605 19627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:08.090986 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:08.091101 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:08.091135 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:08.091151 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336068091151 us; error 0 us; skew 500 ppm
I20251024 20:01:08.091828 19045 webserver.cc:492] Webserver started at http://127.18.153.66:34463/ using document root <none> and password file <none>
I20251024 20:01:08.091913 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:08.091961 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:08.092018 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:08.092288 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/ts-1-root/instance:
uuid: "291a5341be4c4742aa04765e7d101ba9"
format_stamp: "Formatted at 2025-10-24 20:01:08 on dist-test-slave-375r"
I20251024 20:01:08.093068 19622 heartbeater.cc:344] Connected to a master server at 127.18.153.126:35939
I20251024 20:01:08.093135 19622 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:08.093199 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:08.093240 19622 heartbeater.cc:507] Master 127.18.153.126:35939 requested a full tablet report, sending...
I20251024 20:01:08.093529 19493 ts_manager.cc:194] Registered new tserver with Master: 37d31b4fc9a947a0839f629659431c01 (127.18.153.65:38815)
I20251024 20:01:08.094072 19493 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56360
I20251024 20:01:08.094378 19634 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:08.094492 19045 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
I20251024 20:01:08.094542 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/ts-1-root
uuid: "291a5341be4c4742aa04765e7d101ba9"
format_stamp: "Formatted at 2025-10-24 20:01:08 on dist-test-slave-375r"
I20251024 20:01:08.094617 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1761336053687453-19045-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:08.106245 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:08.106474 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:08.106650 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:08.106866 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:08.106899 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:08.106940 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:08.106968 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:08.110460 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.66:44155
I20251024 20:01:08.110529 19697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.66:44155 every 8 connection(s)
I20251024 20:01:08.113859 19698 heartbeater.cc:344] Connected to a master server at 127.18.153.126:35939
I20251024 20:01:08.113940 19698 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:08.114045 19698 heartbeater.cc:507] Master 127.18.153.126:35939 requested a full tablet report, sending...
I20251024 20:01:08.114307 19493 ts_manager.cc:194] Registered new tserver with Master: 291a5341be4c4742aa04765e7d101ba9 (127.18.153.66:44155)
I20251024 20:01:08.114697 19493 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56362
I20251024 20:01:08.115046 19045 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.004375314s
I20251024 20:01:08.119995 19493 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56378:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20251024 20:01:08.120149 19493 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20251024 20:01:08.124365 19662 tablet_service.cc:1505] Processing CreateTablet for tablet bc78801cecb845b7853c506acb5d5975 (DEFAULT_TABLE table=test-table [id=499f01b0d8cf432ca153472d5a5452b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 20:01:08.124504 19662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc78801cecb845b7853c506acb5d5975. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:08.124655 19586 tablet_service.cc:1505] Processing CreateTablet for tablet bc78801cecb845b7853c506acb5d5975 (DEFAULT_TABLE table=test-table [id=499f01b0d8cf432ca153472d5a5452b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 20:01:08.124773 19586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc78801cecb845b7853c506acb5d5975. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:08.125810 19711 tablet_bootstrap.cc:492] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9: Bootstrap starting.
I20251024 20:01:08.125957 19712 tablet_bootstrap.cc:492] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01: Bootstrap starting.
I20251024 20:01:08.126283 19711 tablet_bootstrap.cc:654] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:08.126407 19712 tablet_bootstrap.cc:654] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:08.126844 19711 tablet_bootstrap.cc:492] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9: No bootstrap required, opened a new log
I20251024 20:01:08.126893 19711 ts_tablet_manager.cc:1403] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:08.126947 19712 tablet_bootstrap.cc:492] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01: No bootstrap required, opened a new log
I20251024 20:01:08.126998 19712 ts_tablet_manager.cc:1403] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:08.127101 19711 raft_consensus.cc:359] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:08.127177 19711 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 291a5341be4c4742aa04765e7d101ba9, State: Initialized, Role: FOLLOWER
I20251024 20:01:08.127154 19712 raft_consensus.cc:359] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:08.127210 19711 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:08.127228 19712 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Initialized, Role: FOLLOWER
I20251024 20:01:08.127281 19712 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:08.127393 19711 ts_tablet_manager.cc:1434] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9: Time spent starting tablet: real 0.000s	user 0.001s	sys 0.000s
I20251024 20:01:08.127424 19698 heartbeater.cc:499] Master 127.18.153.126:35939 was elected leader, sending a full tablet report...
I20251024 20:01:08.127592 19622 heartbeater.cc:499] Master 127.18.153.126:35939 was elected leader, sending a full tablet report...
I20251024 20:01:08.127673 19712 ts_tablet_manager.cc:1434] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20251024 20:01:08.134632 19718 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20251024 20:01:08.134684 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 0 FOLLOWER]: Emulating election...
I20251024 20:01:08.134702 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:08.135315 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 1 LEADER]: Becoming Leader. State: Replica: 291a5341be4c4742aa04765e7d101ba9, State: Running, Role: LEADER
I20251024 20:01:08.135418 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
W20251024 20:01:08.135466 19698 heartbeater.cc:646] Failed to heartbeat to 127.18.153.126:35939 (0 consecutive failures): IO error: failing all heartbeats for tests
I20251024 20:01:08.135641 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:08.140951 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20251024 20:01:08.241991 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20251024 20:01:08.443111 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20251024 20:01:08.702386 19596 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:08.703382 19720 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:01:08.745085 19718 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:08.745198 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 1 FOLLOWER]: Emulating election...
I20251024 20:01:08.745244 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:01:08.746230 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 2 LEADER]: Becoming Leader. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:08.746328 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:08.746555 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:08.747218 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 1.1
I20251024 20:01:08.848286 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 1.1
I20251024 20:01:09.049422 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 1.1
I20251024 20:01:09.212595 19596 raft_consensus.cc:1240] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 2 LEADER]: Rejecting Update request from peer 291a5341be4c4742aa04765e7d101ba9 for earlier term 1. Current term is 2. Ops: []
I20251024 20:01:09.212869 19728 consensus_queue.cc:1059] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251024 20:01:09.213028 19728 consensus_queue.cc:1243] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Peer 37d31b4fc9a947a0839f629659431c01 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251024 20:01:09.213094 19729 raft_consensus.cc:3055] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 1 LEADER]: Stepping down as leader of term 1
I20251024 20:01:09.213129 19729 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 291a5341be4c4742aa04765e7d101ba9, State: Running, Role: LEADER
I20251024 20:01:09.213178 19729 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:09.213248 19729 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:01:09.262921 19672 raft_consensus.cc:1275] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 2 FOLLOWER]: Refusing update from remote peer 37d31b4fc9a947a0839f629659431c01: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251024 20:01:09.263206 19730 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251024 20:01:09.351590 19718 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:09.351668 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 2 LEADER]: Emulating election...
I20251024 20:01:09.351689 19718 raft_consensus.cc:3055] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 2 LEADER]: Stepping down as leader of term 2
I20251024 20:01:09.351701 19718 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:09.351744 19718 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:09.351836 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:01:09.352494 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 3 LEADER]: Becoming Leader. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:09.352577 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:09.352736 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:09.353396 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 2.2
I20251024 20:01:09.454541 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 2.2
I20251024 20:01:09.655675 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 2.2
I20251024 20:01:09.943590 19672 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:01:09.944581 19672 raft_consensus.cc:1275] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 3 FOLLOWER]: Refusing update from remote peer 37d31b4fc9a947a0839f629659431c01: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251024 20:01:09.944868 19736 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:01:09.957674 19718 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20251024 20:01:09.957747 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 3 FOLLOWER]: Emulating election...
I20251024 20:01:09.957780 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 3 FOLLOWER]: Advancing to term 4
I20251024 20:01:09.958423 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 4 LEADER]: Becoming Leader. State: Replica: 291a5341be4c4742aa04765e7d101ba9, State: Running, Role: LEADER
I20251024 20:01:09.958508 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:09.958659 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:09.959254 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20251024 20:01:10.060267 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20251024 20:01:10.261518 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20251024 20:01:10.557716 19672 raft_consensus.cc:1240] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 4 LEADER]: Rejecting Update request from peer 37d31b4fc9a947a0839f629659431c01 for earlier term 3. Current term is 4. Ops: []
I20251024 20:01:10.558045 19741 consensus_queue.cc:1059] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251024 20:01:10.558223 19741 consensus_queue.cc:1243] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Peer 291a5341be4c4742aa04765e7d101ba9 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20251024 20:01:10.558275 19741 raft_consensus.cc:3055] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 3 LEADER]: Stepping down as leader of term 3
I20251024 20:01:10.558326 19741 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:10.558380 19741 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:10.558460 19741 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 3 FOLLOWER]: Advancing to term 4
I20251024 20:01:10.562505 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20251024 20:01:10.566155 19596 raft_consensus.cc:1275] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 4 FOLLOWER]: Refusing update from remote peer 291a5341be4c4742aa04765e7d101ba9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251024 20:01:10.566404 19743 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251024 20:01:10.964840 19718 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20251024 20:01:10.964919 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 4 LEADER]: Emulating election...
I20251024 20:01:10.964942 19718 raft_consensus.cc:3055] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 4 LEADER]: Stepping down as leader of term 4
I20251024 20:01:10.964954 19718 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 291a5341be4c4742aa04765e7d101ba9, State: Running, Role: LEADER
I20251024 20:01:10.965001 19718 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:10.965099 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 4 FOLLOWER]: Advancing to term 5
I20251024 20:01:10.965788 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 5 LEADER]: Becoming Leader. State: Replica: 291a5341be4c4742aa04765e7d101ba9, State: Running, Role: LEADER
I20251024 20:01:10.965848 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:10.966004 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:10.966691 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 4.4 5.5
I20251024 20:01:11.067735 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 4.4 5.5
I20251024 20:01:11.268920 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 4.4 5.5
I20251024 20:01:11.569854 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 4.4 5.5
I20251024 20:01:11.574178 19596 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 4 FOLLOWER]: Advancing to term 5
I20251024 20:01:11.574980 19596 raft_consensus.cc:1275] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 5 FOLLOWER]: Refusing update from remote peer 291a5341be4c4742aa04765e7d101ba9: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251024 20:01:11.575284 19747 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251024 20:01:11.972113 19718 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20251024 20:01:11.972196 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 5 LEADER]: Emulating election...
I20251024 20:01:11.972231 19718 raft_consensus.cc:3055] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 5 LEADER]: Stepping down as leader of term 5
I20251024 20:01:11.972251 19718 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 291a5341be4c4742aa04765e7d101ba9, State: Running, Role: LEADER
I20251024 20:01:11.972302 19718 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:11.972396 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 5 FOLLOWER]: Advancing to term 6
I20251024 20:01:11.973101 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 6 LEADER]: Becoming Leader. State: Replica: 291a5341be4c4742aa04765e7d101ba9, State: Running, Role: LEADER
I20251024 20:01:11.973182 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:11.973378 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:11.974048 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 5.5 6.6
I20251024 20:01:12.075047 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 5.5 6.6
I20251024 20:01:12.276019 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 5.5 6.6
I20251024 20:01:12.520419 19596 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 5 FOLLOWER]: Advancing to term 6
I20251024 20:01:12.521109 19596 raft_consensus.cc:1275] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 6 FOLLOWER]: Refusing update from remote peer 291a5341be4c4742aa04765e7d101ba9: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20251024 20:01:12.521373 19753 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251024 20:01:12.578083 19718 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:12.578166 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 6 FOLLOWER]: Emulating election...
I20251024 20:01:12.578199 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 6 FOLLOWER]: Advancing to term 7
I20251024 20:01:12.578851 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 7 LEADER]: Becoming Leader. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:12.578917 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:12.579120 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:12.579754 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 7.7 6.6
I20251024 20:01:12.680676 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 7.7 6.6
I20251024 20:01:12.881616 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 7.7 6.6
I20251024 20:01:13.107764 19672 raft_consensus.cc:3055] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 6 LEADER]: Stepping down as leader of term 6
I20251024 20:01:13.107846 19672 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 291a5341be4c4742aa04765e7d101ba9, State: Running, Role: LEADER
I20251024 20:01:13.107877 19672 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:13.107976 19672 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 6 FOLLOWER]: Advancing to term 7
I20251024 20:01:13.108655 19672 raft_consensus.cc:1275] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 7 FOLLOWER]: Refusing update from remote peer 37d31b4fc9a947a0839f629659431c01: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20251024 20:01:13.108943 19758 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251024 20:01:13.183632 19718 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:13.183712 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 7 LEADER]: Emulating election...
I20251024 20:01:13.183734 19718 raft_consensus.cc:3055] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 7 LEADER]: Stepping down as leader of term 7
I20251024 20:01:13.183746 19718 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:13.183789 19718 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:13.183887 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 7 FOLLOWER]: Advancing to term 8
I20251024 20:01:13.184553 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 8 LEADER]: Becoming Leader. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:13.184628 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:13.184787 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:13.185343 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 8.8 7.7
I20251024 20:01:13.286360 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 8.8 7.7
I20251024 20:01:13.487403 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 8.8 7.7
I20251024 20:01:13.633960 19672 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 7 FOLLOWER]: Advancing to term 8
I20251024 20:01:13.634797 19672 raft_consensus.cc:1275] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 8 FOLLOWER]: Refusing update from remote peer 37d31b4fc9a947a0839f629659431c01: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20251024 20:01:13.635061 19758 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251024 20:01:13.789593 19718 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:13.789670 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 8 LEADER]: Emulating election...
I20251024 20:01:13.789695 19718 raft_consensus.cc:3055] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 8 LEADER]: Stepping down as leader of term 8
I20251024 20:01:13.789707 19718 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:13.789744 19718 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:13.789837 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 8 FOLLOWER]: Advancing to term 9
I20251024 20:01:13.790483 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 9 LEADER]: Becoming Leader. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:13.790541 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:13.790706 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:13.791347 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 9.9 8.8
I20251024 20:01:13.892441 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 9.9 8.8
I20251024 20:01:14.093569 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 9.9 8.8
I20251024 20:01:14.360416 19672 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 8 FOLLOWER]: Advancing to term 9
I20251024 20:01:14.361178 19672 raft_consensus.cc:1275] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 9 FOLLOWER]: Refusing update from remote peer 37d31b4fc9a947a0839f629659431c01: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251024 20:01:14.361469 19767 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251024 20:01:14.395709 19718 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20251024 20:01:14.395787 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 9 FOLLOWER]: Emulating election...
I20251024 20:01:14.395809 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 9 FOLLOWER]: Advancing to term 10
I20251024 20:01:14.396469 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 10 LEADER]: Becoming Leader. State: Replica: 291a5341be4c4742aa04765e7d101ba9, State: Running, Role: LEADER
I20251024 20:01:14.396552 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:14.396724 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:14.397344 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 9.9 10.10
I20251024 20:01:14.498335 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 9.9 10.10
I20251024 20:01:14.699380 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 9.9 10.10
I20251024 20:01:14.809571 19672 raft_consensus.cc:1240] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 10 LEADER]: Rejecting Update request from peer 37d31b4fc9a947a0839f629659431c01 for earlier term 9. Current term is 10. Ops: []
I20251024 20:01:14.809892 19769 consensus_queue.cc:1059] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20251024 20:01:14.810045 19769 consensus_queue.cc:1243] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Peer 291a5341be4c4742aa04765e7d101ba9 log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20251024 20:01:14.810178 19767 raft_consensus.cc:3055] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 9 LEADER]: Stepping down as leader of term 9
I20251024 20:01:14.810225 19767 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:14.810266 19767 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:14.810346 19767 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 9 FOLLOWER]: Advancing to term 10
I20251024 20:01:15.000483 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 9.9 10.10
I20251024 20:01:15.018766 19596 raft_consensus.cc:1275] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 10 FOLLOWER]: Refusing update from remote peer 291a5341be4c4742aa04765e7d101ba9: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20251024 20:01:15.019084 19772 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20251024 20:01:15.402792 19718 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:15.402869 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 10 FOLLOWER]: Emulating election...
I20251024 20:01:15.402892 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 10 FOLLOWER]: Advancing to term 11
I20251024 20:01:15.403638 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 11 LEADER]: Becoming Leader. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:15.403725 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:15.403903 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:15.404619 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 11.11 10.10
I20251024 20:01:15.430760 19596 raft_consensus.cc:1240] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 11 LEADER]: Rejecting Update request from peer 291a5341be4c4742aa04765e7d101ba9 for earlier term 10. Current term is 11. Ops: []
I20251024 20:01:15.431077 19772 consensus_queue.cc:1059] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20251024 20:01:15.431195 19772 consensus_queue.cc:1243] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Peer 37d31b4fc9a947a0839f629659431c01 log is divergent from this leader: its last log entry 11.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20251024 20:01:15.431234 19773 raft_consensus.cc:3055] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 10 LEADER]: Stepping down as leader of term 10
I20251024 20:01:15.431281 19773 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 291a5341be4c4742aa04765e7d101ba9, State: Running, Role: LEADER
I20251024 20:01:15.431325 19773 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:15.431428 19773 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 10 FOLLOWER]: Advancing to term 11
I20251024 20:01:15.505640 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 11.11 10.10
I20251024 20:01:15.706840 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 11.11 10.10
I20251024 20:01:15.858733 19672 raft_consensus.cc:1275] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 11 FOLLOWER]: Refusing update from remote peer 37d31b4fc9a947a0839f629659431c01: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20251024 20:01:15.859043 19767 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20251024 20:01:16.009150 19718 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:16.009246 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 11 LEADER]: Emulating election...
I20251024 20:01:16.009274 19718 raft_consensus.cc:3055] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 11 LEADER]: Stepping down as leader of term 11
I20251024 20:01:16.009289 19718 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:16.009322 19718 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:16.009418 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 11 FOLLOWER]: Advancing to term 12
I20251024 20:01:16.010048 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 12 LEADER]: Becoming Leader. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:16.010133 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:16.010304 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:16.010991 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 12.12 11.11
I20251024 20:01:16.112035 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 12.12 11.11
I20251024 20:01:16.313189 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 12.12 11.11
I20251024 20:01:16.516104 19672 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 11 FOLLOWER]: Advancing to term 12
I20251024 20:01:16.516839 19672 raft_consensus.cc:1275] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 12 FOLLOWER]: Refusing update from remote peer 37d31b4fc9a947a0839f629659431c01: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20251024 20:01:16.517125 19779 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20251024 20:01:16.615413 19718 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:16.615489 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 12 LEADER]: Emulating election...
I20251024 20:01:16.615511 19718 raft_consensus.cc:3055] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 12 LEADER]: Stepping down as leader of term 12
I20251024 20:01:16.615525 19718 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:16.615564 19718 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:16.615659 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 12 FOLLOWER]: Advancing to term 13
I20251024 20:01:16.616389 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 13 LEADER]: Becoming Leader. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:16.616464 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:16.616634 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:16.617215 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 13.13 12.12
I20251024 20:01:16.718104 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 13.13 12.12
I20251024 20:01:16.919188 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 13.13 12.12
I20251024 20:01:17.037175 19672 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 12 FOLLOWER]: Advancing to term 13
I20251024 20:01:17.037986 19672 raft_consensus.cc:1275] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 13 FOLLOWER]: Refusing update from remote peer 37d31b4fc9a947a0839f629659431c01: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20251024 20:01:17.038225 19779 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20251024 20:01:17.221295 19718 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20251024 20:01:17.221388 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 13 FOLLOWER]: Emulating election...
I20251024 20:01:17.221417 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 13 FOLLOWER]: Advancing to term 14
I20251024 20:01:17.222257 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 14 LEADER]: Becoming Leader. State: Replica: 291a5341be4c4742aa04765e7d101ba9, State: Running, Role: LEADER
I20251024 20:01:17.222328 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:17.222482 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:17.223131 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 13.13 14.14
I20251024 20:01:17.324045 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 13.13 14.14
I20251024 20:01:17.457557 19672 raft_consensus.cc:1240] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 14 LEADER]: Rejecting Update request from peer 37d31b4fc9a947a0839f629659431c01 for earlier term 13. Current term is 14. Ops: []
I20251024 20:01:17.457836 19785 consensus_queue.cc:1059] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20251024 20:01:17.457952 19785 consensus_queue.cc:1243] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Peer 291a5341be4c4742aa04765e7d101ba9 log is divergent from this leader: its last log entry 14.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20251024 20:01:17.458077 19779 raft_consensus.cc:3055] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 13 LEADER]: Stepping down as leader of term 13
I20251024 20:01:17.458115 19779 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:17.458149 19779 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:17.458220 19779 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 13 FOLLOWER]: Advancing to term 14
I20251024 20:01:17.524961 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 13.13 14.14
I20251024 20:01:17.721835 19596 raft_consensus.cc:1275] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 14 FOLLOWER]: Refusing update from remote peer 291a5341be4c4742aa04765e7d101ba9: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 14 index: 14. (index mismatch)
I20251024 20:01:17.722118 19784 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20251024 20:01:17.826993 19718 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:17.827072 19718 raft_consensus.cc:423] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 14 FOLLOWER]: Emulating election...
I20251024 20:01:17.827100 19718 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 14 FOLLOWER]: Advancing to term 15
I20251024 20:01:17.827790 19718 raft_consensus.cc:697] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 15 LEADER]: Becoming Leader. State: Replica: 37d31b4fc9a947a0839f629659431c01, State: Running, Role: LEADER
I20251024 20:01:17.827870 19718 consensus_queue.cc:237] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 14.14, Last appended by leader: 14, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:17.828055 19718 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:17.828660 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 15.15 14.14
I20251024 20:01:17.929708 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 15.15 14.14
I20251024 20:01:18.130734 19718 cluster_itest_util.cc:257] Not converged past 0 yet: 15.15 14.14
I20251024 20:01:18.185211 19596 raft_consensus.cc:1240] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 15 LEADER]: Rejecting Update request from peer 291a5341be4c4742aa04765e7d101ba9 for earlier term 14. Current term is 15. Ops: []
I20251024 20:01:18.185521 19788 consensus_queue.cc:1059] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 }, Status: INVALID_TERM, Last received: 14.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20251024 20:01:18.185645 19788 consensus_queue.cc:1243] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [LEADER]: Peer 37d31b4fc9a947a0839f629659431c01 log is divergent from this leader: its last log entry 15.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20251024 20:01:18.185685 19784 raft_consensus.cc:3055] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 14 LEADER]: Stepping down as leader of term 14
I20251024 20:01:18.185734 19784 raft_consensus.cc:740] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 291a5341be4c4742aa04765e7d101ba9, State: Running, Role: LEADER
I20251024 20:01:18.185781 19784 consensus_queue.cc:260] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 14.14, Last appended by leader: 14, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 } } peers { permanent_uuid: "37d31b4fc9a947a0839f629659431c01" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 38815 } }
I20251024 20:01:18.185859 19784 raft_consensus.cc:3060] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 14 FOLLOWER]: Advancing to term 15
I20251024 20:01:18.383138 19672 raft_consensus.cc:1275] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 15 FOLLOWER]: Refusing update from remote peer 37d31b4fc9a947a0839f629659431c01: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 15 index: 15. (index mismatch)
I20251024 20:01:18.383468 19791 consensus_queue.cc:1048] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "291a5341be4c4742aa04765e7d101ba9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20251024 20:01:18.432435 19045 tablet_server.cc:178] TabletServer@127.18.153.65:0 shutting down...
I20251024 20:01:18.435012 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:18.435146 19045 tablet_replica.cc:333] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01: stopping tablet replica
I20251024 20:01:18.435204 19045 raft_consensus.cc:2243] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 15 LEADER]: Raft consensus shutting down.
I20251024 20:01:18.435267 19045 raft_consensus.cc:2272] T bc78801cecb845b7853c506acb5d5975 P 37d31b4fc9a947a0839f629659431c01 [term 15 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:18.436921 19045 tablet_server.cc:195] TabletServer@127.18.153.65:0 shutdown complete.
I20251024 20:01:18.437623 19045 tablet_server.cc:178] TabletServer@127.18.153.66:0 shutting down...
I20251024 20:01:18.440598 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:18.440732 19045 tablet_replica.cc:333] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9: stopping tablet replica
I20251024 20:01:18.440793 19045 raft_consensus.cc:2243] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 15 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:18.440847 19045 raft_consensus.cc:2272] T bc78801cecb845b7853c506acb5d5975 P 291a5341be4c4742aa04765e7d101ba9 [term 15 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:18.442389 19045 tablet_server.cc:195] TabletServer@127.18.153.66:0 shutdown complete.
I20251024 20:01:18.443034 19045 master.cc:561] Master@127.18.153.126:35939 shutting down...
I20251024 20:01:18.445298 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:01:18.445380 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:18.445401 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P e2c7f49d524a4d5badb90f6ebc2aa5c3: stopping tablet replica
I20251024 20:01:18.456835 19045 master.cc:583] Master@127.18.153.126:35939 shutdown complete.
[       OK ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (10442 ms)
[ RUN      ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20251024 20:01:18.461159 19045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.153.126:42613
I20251024 20:01:18.461310 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:18.462576 19800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:18.462637 19799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:18.462731 19802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:18.462738 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:18.462882 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:18.462915 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:18.462941 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336078462942 us; error 0 us; skew 500 ppm
I20251024 20:01:18.463630 19045 webserver.cc:492] Webserver started at http://127.18.153.126:39813/ using document root <none> and password file <none>
I20251024 20:01:18.463712 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:18.463762 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:18.463819 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:18.464107 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/master-0-root/instance:
uuid: "a1a2fc123c7048748b1625032150c248"
format_stamp: "Formatted at 2025-10-24 20:01:18 on dist-test-slave-375r"
I20251024 20:01:18.464953 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:18.465418 19807 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:18.465550 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:18.465590 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/master-0-root
uuid: "a1a2fc123c7048748b1625032150c248"
format_stamp: "Formatted at 2025-10-24 20:01:18 on dist-test-slave-375r"
I20251024 20:01:18.465641 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:18.480722 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:18.480974 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:18.483512 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.126:42613
I20251024 20:01:18.484781 19859 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.126:42613 every 8 connection(s)
I20251024 20:01:18.484898 19860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:18.485867 19860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248: Bootstrap starting.
I20251024 20:01:18.486158 19860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:18.486640 19860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248: No bootstrap required, opened a new log
I20251024 20:01:18.486770 19860 raft_consensus.cc:359] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a2fc123c7048748b1625032150c248" member_type: VOTER }
I20251024 20:01:18.486820 19860 raft_consensus.cc:385] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:18.486841 19860 raft_consensus.cc:740] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a2fc123c7048748b1625032150c248, State: Initialized, Role: FOLLOWER
I20251024 20:01:18.486905 19860 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a2fc123c7048748b1625032150c248" member_type: VOTER }
I20251024 20:01:18.486946 19860 raft_consensus.cc:399] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:01:18.486976 19860 raft_consensus.cc:493] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:01:18.487007 19860 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:18.487504 19860 raft_consensus.cc:515] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a2fc123c7048748b1625032150c248" member_type: VOTER }
I20251024 20:01:18.487560 19860 leader_election.cc:304] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1a2fc123c7048748b1625032150c248; no voters: 
I20251024 20:01:18.487665 19860 leader_election.cc:290] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:01:18.487747 19863 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:01:18.487807 19860 sys_catalog.cc:565] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:01:18.487900 19863 raft_consensus.cc:697] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [term 1 LEADER]: Becoming Leader. State: Replica: a1a2fc123c7048748b1625032150c248, State: Running, Role: LEADER
I20251024 20:01:18.487972 19863 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a2fc123c7048748b1625032150c248" member_type: VOTER }
I20251024 20:01:18.488227 19864 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1a2fc123c7048748b1625032150c248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a2fc123c7048748b1625032150c248" member_type: VOTER } }
I20251024 20:01:18.488245 19865 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1a2fc123c7048748b1625032150c248. Latest consensus state: current_term: 1 leader_uuid: "a1a2fc123c7048748b1625032150c248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a2fc123c7048748b1625032150c248" member_type: VOTER } }
I20251024 20:01:18.488315 19864 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:18.488323 19865 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:18.488577 19870 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:01:18.488708 19870 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:01:18.489250 19870 catalog_manager.cc:1357] Generated new cluster ID: b285ac2443844ca1b389566668650649
I20251024 20:01:18.489246 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251024 20:01:18.489320 19870 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:01:18.510720 19870 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 20:01:18.511130 19870 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:01:18.518642 19870 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248: Generated new TSK 0
I20251024 20:01:18.518733 19870 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:01:18.521085 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:18.522368 19884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:18.522426 19045 server_base.cc:1047] running on GCE node
W20251024 20:01:18.522346 19881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:18.522333 19882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:18.522619 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:18.522655 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:18.522667 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336078522668 us; error 0 us; skew 500 ppm
I20251024 20:01:18.523234 19045 webserver.cc:492] Webserver started at http://127.18.153.65:37061/ using document root <none> and password file <none>
I20251024 20:01:18.523304 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:18.523339 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:18.523422 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:18.523716 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/ts-0-root/instance:
uuid: "ae01638c33de46bbb262830121ebd827"
format_stamp: "Formatted at 2025-10-24 20:01:18 on dist-test-slave-375r"
I20251024 20:01:18.524567 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:18.524992 19889 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:18.525127 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:18.525184 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "ae01638c33de46bbb262830121ebd827"
format_stamp: "Formatted at 2025-10-24 20:01:18 on dist-test-slave-375r"
I20251024 20:01:18.525224 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:18.536401 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:18.536623 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:18.536751 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:18.536948 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:18.536978 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:18.537000 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:18.537029 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:18.539873 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:45413
I20251024 20:01:18.539908 19952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:45413 every 8 connection(s)
I20251024 20:01:18.540278 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:18.541448 19958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:18.541507 19957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:18.541813 19045 server_base.cc:1047] running on GCE node
W20251024 20:01:18.541514 19960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:18.542302 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:18.542341 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:18.542354 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336078542355 us; error 0 us; skew 500 ppm
I20251024 20:01:18.543005 19045 webserver.cc:492] Webserver started at http://127.18.153.66:40623/ using document root <none> and password file <none>
I20251024 20:01:18.543128 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:18.543190 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:18.543236 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:18.543576 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/ts-1-root/instance:
uuid: "d8cf8301eb9f4f39bebf73381ad292d9"
format_stamp: "Formatted at 2025-10-24 20:01:18 on dist-test-slave-375r"
I20251024 20:01:18.543799 19953 heartbeater.cc:344] Connected to a master server at 127.18.153.126:42613
I20251024 20:01:18.543869 19953 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:18.543984 19953 heartbeater.cc:507] Master 127.18.153.126:42613 requested a full tablet report, sending...
I20251024 20:01:18.544201 19824 ts_manager.cc:194] Registered new tserver with Master: ae01638c33de46bbb262830121ebd827 (127.18.153.65:45413)
I20251024 20:01:18.544626 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:18.544746 19824 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57886
I20251024 20:01:18.545143 19965 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:18.545250 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.001s
I20251024 20:01:18.545298 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/ts-1-root
uuid: "d8cf8301eb9f4f39bebf73381ad292d9"
format_stamp: "Formatted at 2025-10-24 20:01:18 on dist-test-slave-375r"
I20251024 20:01:18.545346 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1761336053687453-19045-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:18.596666 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:18.596902 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:18.597045 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:18.597260 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:18.597290 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:18.597316 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:18.597345 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:18.600346 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.66:36115
I20251024 20:01:18.600422 20028 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.66:36115 every 8 connection(s)
I20251024 20:01:18.603469 20029 heartbeater.cc:344] Connected to a master server at 127.18.153.126:42613
I20251024 20:01:18.603531 20029 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:18.603638 20029 heartbeater.cc:507] Master 127.18.153.126:42613 requested a full tablet report, sending...
I20251024 20:01:18.603889 19824 ts_manager.cc:194] Registered new tserver with Master: d8cf8301eb9f4f39bebf73381ad292d9 (127.18.153.66:36115)
I20251024 20:01:18.604233 19824 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57898
I20251024 20:01:18.604940 19045 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.004395271s
I20251024 20:01:18.609401 19824 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57902:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20251024 20:01:18.609541 19824 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20251024 20:01:18.614619 19917 tablet_service.cc:1505] Processing CreateTablet for tablet f019f53ec78d4a11b8a850d1e7fab53f (DEFAULT_TABLE table=test-table [id=cb7a4fc88e544012a42b6026ac8936e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 20:01:18.614619 19993 tablet_service.cc:1505] Processing CreateTablet for tablet f019f53ec78d4a11b8a850d1e7fab53f (DEFAULT_TABLE table=test-table [id=cb7a4fc88e544012a42b6026ac8936e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 20:01:18.614805 19993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f019f53ec78d4a11b8a850d1e7fab53f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:18.614804 19917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f019f53ec78d4a11b8a850d1e7fab53f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:18.616405 20043 tablet_bootstrap.cc:492] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9: Bootstrap starting.
I20251024 20:01:18.616559 20042 tablet_bootstrap.cc:492] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827: Bootstrap starting.
I20251024 20:01:18.617033 20042 tablet_bootstrap.cc:654] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:18.617048 20043 tablet_bootstrap.cc:654] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:18.617595 20043 tablet_bootstrap.cc:492] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9: No bootstrap required, opened a new log
I20251024 20:01:18.617626 20042 tablet_bootstrap.cc:492] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827: No bootstrap required, opened a new log
I20251024 20:01:18.617638 20043 ts_tablet_manager.cc:1403] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:18.617704 20042 ts_tablet_manager.cc:1403] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:18.617825 20043 raft_consensus.cc:359] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:18.617883 20043 raft_consensus.cc:740] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8cf8301eb9f4f39bebf73381ad292d9, State: Initialized, Role: FOLLOWER
I20251024 20:01:18.617864 20042 raft_consensus.cc:359] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:18.617933 20043 consensus_queue.cc:260] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:18.617940 20042 raft_consensus.cc:740] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae01638c33de46bbb262830121ebd827, State: Initialized, Role: FOLLOWER
I20251024 20:01:18.618011 20042 consensus_queue.cc:260] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:18.618124 20043 ts_tablet_manager.cc:1434] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9: Time spent starting tablet: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:18.618171 20042 ts_tablet_manager.cc:1434] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827: Time spent starting tablet: real 0.000s	user 0.001s	sys 0.000s
I20251024 20:01:18.618196 19953 heartbeater.cc:499] Master 127.18.153.126:42613 was elected leader, sending a full tablet report...
I20251024 20:01:18.618180 20029 heartbeater.cc:499] Master 127.18.153.126:42613 was elected leader, sending a full tablet report...
I20251024 20:01:18.622494 19045 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20251024 20:01:18.622538 19045 raft_consensus.cc:423] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 0 FOLLOWER]: Emulating election...
I20251024 20:01:18.622565 19045 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:18.623142 19045 raft_consensus.cc:697] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 1 LEADER]: Becoming Leader. State: Replica: d8cf8301eb9f4f39bebf73381ad292d9, State: Running, Role: LEADER
I20251024 20:01:18.623193 19045 consensus_queue.cc:237] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:18.623468 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:18.628567 19045 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 1.1
I20251024 20:01:18.729593 19045 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 1.1
I20251024 20:01:18.930784 19045 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 1.1
I20251024 20:01:19.123179 19927 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:19.124207 20046 consensus_queue.cc:1048] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:01:19.232208 19045 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20251024 20:01:19.232282 19045 raft_consensus.cc:423] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 1 LEADER]: Emulating election...
I20251024 20:01:19.232303 19045 raft_consensus.cc:3055] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 1 LEADER]: Stepping down as leader of term 1
I20251024 20:01:19.232316 19045 raft_consensus.cc:740] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8cf8301eb9f4f39bebf73381ad292d9, State: Running, Role: LEADER
I20251024 20:01:19.232360 19045 consensus_queue.cc:260] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:19.232430 19045 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:01:19.233017 19045 raft_consensus.cc:697] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 2 LEADER]: Becoming Leader. State: Replica: d8cf8301eb9f4f39bebf73381ad292d9, State: Running, Role: LEADER
I20251024 20:01:19.233069 19045 consensus_queue.cc:237] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:19.233203 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:19.233801 19045 cluster_itest_util.cc:257] Not converged past 2 yet: 1.1 2.2
I20251024 20:01:19.334916 19045 cluster_itest_util.cc:257] Not converged past 2 yet: 1.1 2.2
I20251024 20:01:19.536146 19045 cluster_itest_util.cc:257] Not converged past 2 yet: 1.1 2.2
I20251024 20:01:19.662961 19927 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:01:19.663753 19927 raft_consensus.cc:1275] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 2 FOLLOWER]: Refusing update from remote peer d8cf8301eb9f4f39bebf73381ad292d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251024 20:01:19.664005 20046 consensus_queue.cc:1048] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251024 20:01:19.837584 19045 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:19.837653 19045 raft_consensus.cc:423] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 2 FOLLOWER]: Emulating election...
I20251024 20:01:19.837672 19045 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:01:19.838321 19045 raft_consensus.cc:697] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 3 LEADER]: Becoming Leader. State: Replica: ae01638c33de46bbb262830121ebd827, State: Running, Role: LEADER
I20251024 20:01:19.838394 19045 consensus_queue.cc:237] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:19.838575 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:19.839126 19045 cluster_itest_util.cc:257] Not converged past 3 yet: 3.3 2.2
I20251024 20:01:19.940230 19045 cluster_itest_util.cc:257] Not converged past 3 yet: 3.3 2.2
I20251024 20:01:20.064011 19927 raft_consensus.cc:1240] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 3 LEADER]: Rejecting Update request from peer d8cf8301eb9f4f39bebf73381ad292d9 for earlier term 2. Current term is 3. Ops: []
I20251024 20:01:20.064312 20057 consensus_queue.cc:1059] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:01:20.064421 20057 consensus_queue.cc:1243] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [LEADER]: Peer ae01638c33de46bbb262830121ebd827 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251024 20:01:20.064445 20057 raft_consensus.cc:3055] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 2 LEADER]: Stepping down as leader of term 2
I20251024 20:01:20.064476 20057 raft_consensus.cc:740] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8cf8301eb9f4f39bebf73381ad292d9, State: Running, Role: LEADER
I20251024 20:01:20.064517 20057 consensus_queue.cc:260] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:20.064579 20057 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:01:20.141379 19045 cluster_itest_util.cc:257] Not converged past 3 yet: 3.3 2.2
I20251024 20:01:20.371027 20003 raft_consensus.cc:1275] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 3 FOLLOWER]: Refusing update from remote peer ae01638c33de46bbb262830121ebd827: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251024 20:01:20.371335 20060 consensus_queue.cc:1048] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:01:20.442602 19045 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:20.442695 19045 raft_consensus.cc:423] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 3 LEADER]: Emulating election...
I20251024 20:01:20.442718 19045 raft_consensus.cc:3055] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 3 LEADER]: Stepping down as leader of term 3
I20251024 20:01:20.442730 19045 raft_consensus.cc:740] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ae01638c33de46bbb262830121ebd827, State: Running, Role: LEADER
I20251024 20:01:20.442754 19045 consensus_queue.cc:260] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:20.442826 19045 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 3 FOLLOWER]: Advancing to term 4
I20251024 20:01:20.443522 19045 raft_consensus.cc:697] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 4 LEADER]: Becoming Leader. State: Replica: ae01638c33de46bbb262830121ebd827, State: Running, Role: LEADER
I20251024 20:01:20.443578 19045 consensus_queue.cc:237] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:20.443708 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:20.444358 19045 cluster_itest_util.cc:257] Not converged past 4 yet: 4.4 3.3
I20251024 20:01:20.545318 19045 cluster_itest_util.cc:257] Not converged past 4 yet: 4.4 3.3
I20251024 20:01:20.746222 19045 cluster_itest_util.cc:257] Not converged past 4 yet: 4.4 3.3
I20251024 20:01:21.047255 19045 cluster_itest_util.cc:257] Not converged past 4 yet: 4.4 3.3
I20251024 20:01:21.051914 20003 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 3 FOLLOWER]: Advancing to term 4
I20251024 20:01:21.054565 20003 raft_consensus.cc:1275] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 4 FOLLOWER]: Refusing update from remote peer ae01638c33de46bbb262830121ebd827: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251024 20:01:21.054875 20066 consensus_queue.cc:1048] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251024 20:01:21.448503 19045 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:21.448580 19045 raft_consensus.cc:423] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 4 LEADER]: Emulating election...
I20251024 20:01:21.448603 19045 raft_consensus.cc:3055] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 4 LEADER]: Stepping down as leader of term 4
I20251024 20:01:21.448618 19045 raft_consensus.cc:740] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ae01638c33de46bbb262830121ebd827, State: Running, Role: LEADER
I20251024 20:01:21.448645 19045 consensus_queue.cc:260] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:21.448724 19045 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 4 FOLLOWER]: Advancing to term 5
I20251024 20:01:21.449374 19045 raft_consensus.cc:697] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 5 LEADER]: Becoming Leader. State: Replica: ae01638c33de46bbb262830121ebd827, State: Running, Role: LEADER
I20251024 20:01:21.449443 19045 consensus_queue.cc:237] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:21.449615 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:21.450194 19045 cluster_itest_util.cc:257] Not converged past 5 yet: 5.5 4.4
I20251024 20:01:21.551087 19045 cluster_itest_util.cc:257] Not converged past 5 yet: 5.5 4.4
I20251024 20:01:21.752208 19045 cluster_itest_util.cc:257] Not converged past 5 yet: 5.5 4.4
I20251024 20:01:22.040794 20003 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 4 FOLLOWER]: Advancing to term 5
I20251024 20:01:22.041551 20003 raft_consensus.cc:1275] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 5 FOLLOWER]: Refusing update from remote peer ae01638c33de46bbb262830121ebd827: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251024 20:01:22.041800 20071 consensus_queue.cc:1048] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251024 20:01:22.053349 19045 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:22.053418 19045 raft_consensus.cc:423] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 5 LEADER]: Emulating election...
I20251024 20:01:22.053448 19045 raft_consensus.cc:3055] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 5 LEADER]: Stepping down as leader of term 5
I20251024 20:01:22.053467 19045 raft_consensus.cc:740] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ae01638c33de46bbb262830121ebd827, State: Running, Role: LEADER
I20251024 20:01:22.053501 19045 consensus_queue.cc:260] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:22.053586 19045 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 5 FOLLOWER]: Advancing to term 6
I20251024 20:01:22.054203 19045 raft_consensus.cc:697] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 6 LEADER]: Becoming Leader. State: Replica: ae01638c33de46bbb262830121ebd827, State: Running, Role: LEADER
I20251024 20:01:22.054257 19045 consensus_queue.cc:237] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:22.054402 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:22.055047 19045 cluster_itest_util.cc:257] Not converged past 6 yet: 6.6 5.5
I20251024 20:01:22.156262 19045 cluster_itest_util.cc:257] Not converged past 6 yet: 6.6 5.5
I20251024 20:01:22.357303 19045 cluster_itest_util.cc:257] Not converged past 6 yet: 6.6 5.5
I20251024 20:01:22.596946 20003 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 5 FOLLOWER]: Advancing to term 6
I20251024 20:01:22.597710 20003 raft_consensus.cc:1275] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 6 FOLLOWER]: Refusing update from remote peer ae01638c33de46bbb262830121ebd827: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20251024 20:01:22.597960 20076 consensus_queue.cc:1048] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251024 20:01:22.658706 19045 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:22.658777 19045 raft_consensus.cc:423] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 6 LEADER]: Emulating election...
I20251024 20:01:22.658798 19045 raft_consensus.cc:3055] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 6 LEADER]: Stepping down as leader of term 6
I20251024 20:01:22.658811 19045 raft_consensus.cc:740] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ae01638c33de46bbb262830121ebd827, State: Running, Role: LEADER
I20251024 20:01:22.658845 19045 consensus_queue.cc:260] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:22.658939 19045 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 6 FOLLOWER]: Advancing to term 7
I20251024 20:01:22.659617 19045 raft_consensus.cc:697] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 7 LEADER]: Becoming Leader. State: Replica: ae01638c33de46bbb262830121ebd827, State: Running, Role: LEADER
I20251024 20:01:22.659682 19045 consensus_queue.cc:237] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:22.659843 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:22.660457 19045 cluster_itest_util.cc:257] Not converged past 7 yet: 7.7 6.6
I20251024 20:01:22.761392 19045 cluster_itest_util.cc:257] Not converged past 7 yet: 7.7 6.6
I20251024 20:01:22.962390 19045 cluster_itest_util.cc:257] Not converged past 7 yet: 7.7 6.6
I20251024 20:01:23.160545 20003 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 6 FOLLOWER]: Advancing to term 7
I20251024 20:01:23.161259 20003 raft_consensus.cc:1275] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 7 FOLLOWER]: Refusing update from remote peer ae01638c33de46bbb262830121ebd827: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20251024 20:01:23.161520 20081 consensus_queue.cc:1048] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251024 20:01:23.263633 19045 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20251024 20:01:23.263705 19045 raft_consensus.cc:423] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 7 FOLLOWER]: Emulating election...
I20251024 20:01:23.263725 19045 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 7 FOLLOWER]: Advancing to term 8
I20251024 20:01:23.264350 19045 raft_consensus.cc:697] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 8 LEADER]: Becoming Leader. State: Replica: d8cf8301eb9f4f39bebf73381ad292d9, State: Running, Role: LEADER
I20251024 20:01:23.264405 19045 consensus_queue.cc:237] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:23.264562 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:23.265131 19045 cluster_itest_util.cc:257] Not converged past 8 yet: 7.7 8.8
I20251024 20:01:23.366087 19045 cluster_itest_util.cc:257] Not converged past 8 yet: 7.7 8.8
I20251024 20:01:23.567142 19045 cluster_itest_util.cc:257] Not converged past 8 yet: 7.7 8.8
I20251024 20:01:23.575978 20003 raft_consensus.cc:1240] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 8 LEADER]: Rejecting Update request from peer ae01638c33de46bbb262830121ebd827 for earlier term 7. Current term is 8. Ops: []
I20251024 20:01:23.576275 20083 consensus_queue.cc:1059] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251024 20:01:23.576386 20083 consensus_queue.cc:1243] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [LEADER]: Peer d8cf8301eb9f4f39bebf73381ad292d9 log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20251024 20:01:23.576427 20081 raft_consensus.cc:3055] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 7 LEADER]: Stepping down as leader of term 7
I20251024 20:01:23.576468 20081 raft_consensus.cc:740] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ae01638c33de46bbb262830121ebd827, State: Running, Role: LEADER
I20251024 20:01:23.576500 20081 consensus_queue.cc:260] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:23.576552 20081 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 7 FOLLOWER]: Advancing to term 8
I20251024 20:01:23.777778 19927 raft_consensus.cc:1275] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 8 FOLLOWER]: Refusing update from remote peer d8cf8301eb9f4f39bebf73381ad292d9: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20251024 20:01:23.778076 20086 consensus_queue.cc:1048] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251024 20:01:23.868153 19045 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20251024 20:01:23.868223 19045 raft_consensus.cc:423] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 8 LEADER]: Emulating election...
I20251024 20:01:23.868243 19045 raft_consensus.cc:3055] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 8 LEADER]: Stepping down as leader of term 8
I20251024 20:01:23.868256 19045 raft_consensus.cc:740] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d8cf8301eb9f4f39bebf73381ad292d9, State: Running, Role: LEADER
I20251024 20:01:23.868284 19045 consensus_queue.cc:260] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:23.868386 19045 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 8 FOLLOWER]: Advancing to term 9
I20251024 20:01:23.869021 19045 raft_consensus.cc:697] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 9 LEADER]: Becoming Leader. State: Replica: d8cf8301eb9f4f39bebf73381ad292d9, State: Running, Role: LEADER
I20251024 20:01:23.869128 19045 consensus_queue.cc:237] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:23.869287 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:23.869895 19045 cluster_itest_util.cc:257] Not converged past 9 yet: 8.8 9.9
I20251024 20:01:23.970937 19045 cluster_itest_util.cc:257] Not converged past 9 yet: 8.8 9.9
I20251024 20:01:24.172163 19045 cluster_itest_util.cc:257] Not converged past 9 yet: 8.8 9.9
I20251024 20:01:24.320117 19927 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 8 FOLLOWER]: Advancing to term 9
I20251024 20:01:24.320899 19927 raft_consensus.cc:1275] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 9 FOLLOWER]: Refusing update from remote peer d8cf8301eb9f4f39bebf73381ad292d9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251024 20:01:24.321187 20086 consensus_queue.cc:1048] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251024 20:01:24.473380 19045 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20251024 20:01:24.473476 19045 raft_consensus.cc:423] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 9 FOLLOWER]: Emulating election...
I20251024 20:01:24.473505 19045 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 9 FOLLOWER]: Advancing to term 10
I20251024 20:01:24.474179 19045 raft_consensus.cc:697] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 10 LEADER]: Becoming Leader. State: Replica: ae01638c33de46bbb262830121ebd827, State: Running, Role: LEADER
I20251024 20:01:24.474247 19045 consensus_queue.cc:237] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:24.474411 19045 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20251024 20:01:24.475129 19045 cluster_itest_util.cc:257] Not converged past 10 yet: 10.10 9.9
I20251024 20:01:24.576171 19045 cluster_itest_util.cc:257] Not converged past 10 yet: 10.10 9.9
I20251024 20:01:24.741154 19927 raft_consensus.cc:1240] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 10 LEADER]: Rejecting Update request from peer d8cf8301eb9f4f39bebf73381ad292d9 for earlier term 9. Current term is 10. Ops: []
I20251024 20:01:24.741389 20091 consensus_queue.cc:1059] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20251024 20:01:24.741515 20091 consensus_queue.cc:1243] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [LEADER]: Peer ae01638c33de46bbb262830121ebd827 log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20251024 20:01:24.741560 20086 raft_consensus.cc:3055] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 9 LEADER]: Stepping down as leader of term 9
I20251024 20:01:24.741638 20086 raft_consensus.cc:740] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d8cf8301eb9f4f39bebf73381ad292d9, State: Running, Role: LEADER
I20251024 20:01:24.741691 20086 consensus_queue.cc:260] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 } } peers { permanent_uuid: "ae01638c33de46bbb262830121ebd827" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45413 } }
I20251024 20:01:24.741791 20086 raft_consensus.cc:3060] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 9 FOLLOWER]: Advancing to term 10
I20251024 20:01:24.777200 19045 cluster_itest_util.cc:257] Not converged past 10 yet: 10.10 9.9
I20251024 20:01:24.980576 20003 raft_consensus.cc:1275] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 10 FOLLOWER]: Refusing update from remote peer ae01638c33de46bbb262830121ebd827: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20251024 20:01:24.980851 20094 consensus_queue.cc:1048] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8cf8301eb9f4f39bebf73381ad292d9" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20251024 20:01:25.078449 19045 tablet_server.cc:178] TabletServer@127.18.153.65:0 shutting down...
I20251024 20:01:25.081243 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:25.081395 19045 tablet_replica.cc:333] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827: stopping tablet replica
I20251024 20:01:25.081452 19045 raft_consensus.cc:2243] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 10 LEADER]: Raft consensus shutting down.
I20251024 20:01:25.081522 19045 raft_consensus.cc:2272] T f019f53ec78d4a11b8a850d1e7fab53f P ae01638c33de46bbb262830121ebd827 [term 10 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:25.093693 19045 tablet_server.cc:195] TabletServer@127.18.153.65:0 shutdown complete.
I20251024 20:01:25.094584 19045 tablet_server.cc:178] TabletServer@127.18.153.66:0 shutting down...
I20251024 20:01:25.097009 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:25.097118 19045 tablet_replica.cc:333] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9: stopping tablet replica
I20251024 20:01:25.097177 19045 raft_consensus.cc:2243] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 10 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:25.097230 19045 raft_consensus.cc:2272] T f019f53ec78d4a11b8a850d1e7fab53f P d8cf8301eb9f4f39bebf73381ad292d9 [term 10 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:25.108932 19045 tablet_server.cc:195] TabletServer@127.18.153.66:0 shutdown complete.
I20251024 20:01:25.109819 19045 master.cc:561] Master@127.18.153.126:42613 shutting down...
I20251024 20:01:25.112234 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:01:25.112311 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:25.112334 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P a1a2fc123c7048748b1625032150c248: stopping tablet replica
I20251024 20:01:25.123827 19045 master.cc:583] Master@127.18.153.126:42613 shutdown complete.
[       OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (6666 ms)
[----------] 3 tests from LeadershipChangeReportingTest (31435 ms total)

[----------] 4 tests from TsTabletManagerITest
[ RUN      ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20251024 20:01:25.128057 19045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.153.126:45119
I20251024 20:01:25.128217 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:25.129595 20102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:25.129582 20105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:25.129741 20103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:25.129912 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:25.130008 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:25.130035 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:25.130059 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336085130058 us; error 0 us; skew 500 ppm
I20251024 20:01:25.130722 19045 webserver.cc:492] Webserver started at http://127.18.153.126:33717/ using document root <none> and password file <none>
I20251024 20:01:25.130820 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:25.130865 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:25.130915 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:25.131207 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/master-0-root/instance:
uuid: "3fce915e481c4f34a63d518c0286e307"
format_stamp: "Formatted at 2025-10-24 20:01:25 on dist-test-slave-375r"
I20251024 20:01:25.132100 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:25.132589 20110 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.132717 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.132766 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/master-0-root
uuid: "3fce915e481c4f34a63d518c0286e307"
format_stamp: "Formatted at 2025-10-24 20:01:25 on dist-test-slave-375r"
I20251024 20:01:25.132807 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:25.143407 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:25.143599 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:25.146112 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.126:45119
I20251024 20:01:25.147260 20162 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.126:45119 every 8 connection(s)
I20251024 20:01:25.147450 20163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:25.148439 20163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307: Bootstrap starting.
I20251024 20:01:25.148710 20163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:25.149169 20163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307: No bootstrap required, opened a new log
I20251024 20:01:25.149274 20163 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fce915e481c4f34a63d518c0286e307" member_type: VOTER }
I20251024 20:01:25.149323 20163 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:25.149338 20163 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fce915e481c4f34a63d518c0286e307, State: Initialized, Role: FOLLOWER
I20251024 20:01:25.149387 20163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fce915e481c4f34a63d518c0286e307" member_type: VOTER }
I20251024 20:01:25.149430 20163 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:01:25.149452 20163 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:01:25.149474 20163 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:25.149941 20163 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fce915e481c4f34a63d518c0286e307" member_type: VOTER }
I20251024 20:01:25.149994 20163 leader_election.cc:304] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fce915e481c4f34a63d518c0286e307; no voters: 
I20251024 20:01:25.150087 20163 leader_election.cc:290] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:01:25.150163 20166 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:01:25.150240 20163 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:01:25.150270 20166 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [term 1 LEADER]: Becoming Leader. State: Replica: 3fce915e481c4f34a63d518c0286e307, State: Running, Role: LEADER
I20251024 20:01:25.150374 20166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fce915e481c4f34a63d518c0286e307" member_type: VOTER }
I20251024 20:01:25.150610 20167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fce915e481c4f34a63d518c0286e307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fce915e481c4f34a63d518c0286e307" member_type: VOTER } }
I20251024 20:01:25.150696 20167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:25.150913 20168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fce915e481c4f34a63d518c0286e307. Latest consensus state: current_term: 1 leader_uuid: "3fce915e481c4f34a63d518c0286e307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fce915e481c4f34a63d518c0286e307" member_type: VOTER } }
I20251024 20:01:25.150943 20172 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:01:25.151017 20168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:25.151176 20172 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:01:25.151618 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251024 20:01:25.151782 20172 catalog_manager.cc:1357] Generated new cluster ID: afcca6bdff34426a974e38f98d5d01af
I20251024 20:01:25.151818 20172 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:01:25.163255 20172 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 20:01:25.163756 20172 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:01:25.168876 20172 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307: Generated new TSK 0
I20251024 20:01:25.168985 20172 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:01:25.183641 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:25.184966 20185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:25.185001 20187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:25.185115 19045 server_base.cc:1047] running on GCE node
W20251024 20:01:25.185118 20184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:25.185391 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:25.185427 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:25.185441 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336085185440 us; error 0 us; skew 500 ppm
I20251024 20:01:25.186036 19045 webserver.cc:492] Webserver started at http://127.18.153.65:36233/ using document root <none> and password file <none>
I20251024 20:01:25.186125 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:25.186163 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:25.186201 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:25.186465 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-0-root/instance:
uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e"
format_stamp: "Formatted at 2025-10-24 20:01:25 on dist-test-slave-375r"
I20251024 20:01:25.187310 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:25.187784 20192 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.187932 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.187979 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e"
format_stamp: "Formatted at 2025-10-24 20:01:25 on dist-test-slave-375r"
I20251024 20:01:25.188020 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:25.196826 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:25.197031 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:25.197165 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:25.197355 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:25.197386 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.197410 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:25.197439 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.200318 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:42755
I20251024 20:01:25.200357 20255 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:42755 every 8 connection(s)
I20251024 20:01:25.200731 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:25.201857 20260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:25.201854 20263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:25.202005 20261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:25.202621 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:25.202740 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:25.202770 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:25.202792 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336085202791 us; error 0 us; skew 500 ppm
I20251024 20:01:25.203462 19045 webserver.cc:492] Webserver started at http://127.18.153.66:46761/ using document root <none> and password file <none>
I20251024 20:01:25.203549 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:25.203603 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:25.203691 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:25.204020 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-1-root/instance:
uuid: "7feabb2b6b524440bcd26b7158d241f2"
format_stamp: "Formatted at 2025-10-24 20:01:25 on dist-test-slave-375r"
I20251024 20:01:25.204918 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:25.205397 20268 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.205538 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.205582 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-1-root
uuid: "7feabb2b6b524440bcd26b7158d241f2"
format_stamp: "Formatted at 2025-10-24 20:01:25 on dist-test-slave-375r"
I20251024 20:01:25.205636 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:25.208379 20256 heartbeater.cc:344] Connected to a master server at 127.18.153.126:45119
I20251024 20:01:25.208439 20256 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:25.208529 20256 heartbeater.cc:507] Master 127.18.153.126:45119 requested a full tablet report, sending...
I20251024 20:01:25.208766 20127 ts_manager.cc:194] Registered new tserver with Master: 9b59e5f6e6bd4a8697ffac00418ffc4e (127.18.153.65:42755)
I20251024 20:01:25.209280 20127 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36858
I20251024 20:01:25.228827 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:25.229063 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:25.229231 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:25.229449 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:25.229480 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.229516 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:25.229542 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.232565 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.66:42967
I20251024 20:01:25.232635 20331 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.66:42967 every 8 connection(s)
I20251024 20:01:25.232947 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:25.234177 20336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:25.234251 20335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:25.234426 19045 server_base.cc:1047] running on GCE node
W20251024 20:01:25.235399 20338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:25.235538 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:25.235582 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:25.235600 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336085235601 us; error 0 us; skew 500 ppm
I20251024 20:01:25.236183 19045 webserver.cc:492] Webserver started at http://127.18.153.67:42865/ using document root <none> and password file <none>
I20251024 20:01:25.236271 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:25.236312 20332 heartbeater.cc:344] Connected to a master server at 127.18.153.126:45119
I20251024 20:01:25.236322 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:25.236392 20332 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:25.236411 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:25.236505 20332 heartbeater.cc:507] Master 127.18.153.126:45119 requested a full tablet report, sending...
I20251024 20:01:25.236743 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-2-root/instance:
uuid: "ac54a7903d8e4405ab898e38f557aad1"
format_stamp: "Formatted at 2025-10-24 20:01:25 on dist-test-slave-375r"
I20251024 20:01:25.236756 20127 ts_manager.cc:194] Registered new tserver with Master: 7feabb2b6b524440bcd26b7158d241f2 (127.18.153.66:42967)
I20251024 20:01:25.237128 20127 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36862
I20251024 20:01:25.237758 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:25.238217 20343 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.238349 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.238394 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-2-root
uuid: "ac54a7903d8e4405ab898e38f557aad1"
format_stamp: "Formatted at 2025-10-24 20:01:25 on dist-test-slave-375r"
I20251024 20:01:25.238446 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1761336053687453-19045-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:25.245146 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:25.245417 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:25.245596 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:25.245797 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:25.245828 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.245862 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:25.245888 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.249219 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.67:45671
I20251024 20:01:25.249285 20406 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.67:45671 every 8 connection(s)
I20251024 20:01:25.252516 20407 heartbeater.cc:344] Connected to a master server at 127.18.153.126:45119
I20251024 20:01:25.252580 20407 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:25.252683 20407 heartbeater.cc:507] Master 127.18.153.126:45119 requested a full tablet report, sending...
I20251024 20:01:25.252923 20127 ts_manager.cc:194] Registered new tserver with Master: ac54a7903d8e4405ab898e38f557aad1 (127.18.153.67:45671)
I20251024 20:01:25.253254 20127 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36874
I20251024 20:01:25.253829 19045 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004418622s
I20251024 20:01:25.258354 20127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36882:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
  }
}
W20251024 20:01:25.258476 20127 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 20:01:25.263311 20220 tablet_service.cc:1505] Processing CreateTablet for tablet 35233874510d490b9db81f13f82965f3 (DEFAULT_TABLE table=test-table [id=2a68c79c12284691ba8f287f74731e65]), partition=
I20251024 20:01:25.263303 20371 tablet_service.cc:1505] Processing CreateTablet for tablet 35233874510d490b9db81f13f82965f3 (DEFAULT_TABLE table=test-table [id=2a68c79c12284691ba8f287f74731e65]), partition=
I20251024 20:01:25.263517 20220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35233874510d490b9db81f13f82965f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:25.263543 20371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35233874510d490b9db81f13f82965f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:25.264779 20422 tablet_bootstrap.cc:492] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1: Bootstrap starting.
I20251024 20:01:25.264938 20296 tablet_service.cc:1505] Processing CreateTablet for tablet 35233874510d490b9db81f13f82965f3 (DEFAULT_TABLE table=test-table [id=2a68c79c12284691ba8f287f74731e65]), partition=
I20251024 20:01:25.265041 20296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35233874510d490b9db81f13f82965f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:25.265204 20423 tablet_bootstrap.cc:492] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e: Bootstrap starting.
I20251024 20:01:25.265292 20422 tablet_bootstrap.cc:654] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:25.265815 20423 tablet_bootstrap.cc:654] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:25.265959 20422 tablet_bootstrap.cc:492] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1: No bootstrap required, opened a new log
I20251024 20:01:25.266010 20422 ts_tablet_manager.cc:1403] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:25.266237 20422 raft_consensus.cc:359] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42755 } } peers { permanent_uuid: "7feabb2b6b524440bcd26b7158d241f2" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42967 } } peers { permanent_uuid: "ac54a7903d8e4405ab898e38f557aad1" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45671 } }
I20251024 20:01:25.266324 20422 raft_consensus.cc:385] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:25.266350 20422 raft_consensus.cc:740] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac54a7903d8e4405ab898e38f557aad1, State: Initialized, Role: FOLLOWER
I20251024 20:01:25.266423 20423 tablet_bootstrap.cc:492] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e: No bootstrap required, opened a new log
I20251024 20:01:25.266469 20423 ts_tablet_manager.cc:1403] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:25.266448 20422 consensus_queue.cc:260] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42755 } } peers { permanent_uuid: "7feabb2b6b524440bcd26b7158d241f2" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42967 } } peers { permanent_uuid: "ac54a7903d8e4405ab898e38f557aad1" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45671 } }
I20251024 20:01:25.266630 20423 raft_consensus.cc:359] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42755 } } peers { permanent_uuid: "7feabb2b6b524440bcd26b7158d241f2" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42967 } } peers { permanent_uuid: "ac54a7903d8e4405ab898e38f557aad1" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45671 } }
I20251024 20:01:25.266693 20423 raft_consensus.cc:385] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:25.266702 20407 heartbeater.cc:499] Master 127.18.153.126:45119 was elected leader, sending a full tablet report...
I20251024 20:01:25.266719 20423 raft_consensus.cc:740] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b59e5f6e6bd4a8697ffac00418ffc4e, State: Initialized, Role: FOLLOWER
I20251024 20:01:25.266777 20426 tablet_bootstrap.cc:492] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2: Bootstrap starting.
I20251024 20:01:25.266851 20423 consensus_queue.cc:260] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42755 } } peers { permanent_uuid: "7feabb2b6b524440bcd26b7158d241f2" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42967 } } peers { permanent_uuid: "ac54a7903d8e4405ab898e38f557aad1" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45671 } }
I20251024 20:01:25.266839 20422 ts_tablet_manager.cc:1434] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:25.267077 20423 ts_tablet_manager.cc:1434] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:25.267206 20256 heartbeater.cc:499] Master 127.18.153.126:45119 was elected leader, sending a full tablet report...
I20251024 20:01:25.267257 20426 tablet_bootstrap.cc:654] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:25.267897 20426 tablet_bootstrap.cc:492] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2: No bootstrap required, opened a new log
I20251024 20:01:25.267938 20426 ts_tablet_manager.cc:1403] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:25.268040 20426 raft_consensus.cc:359] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42755 } } peers { permanent_uuid: "7feabb2b6b524440bcd26b7158d241f2" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42967 } } peers { permanent_uuid: "ac54a7903d8e4405ab898e38f557aad1" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45671 } }
I20251024 20:01:25.268083 20426 raft_consensus.cc:385] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:25.268096 20426 raft_consensus.cc:740] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7feabb2b6b524440bcd26b7158d241f2, State: Initialized, Role: FOLLOWER
I20251024 20:01:25.268131 20426 consensus_queue.cc:260] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42755 } } peers { permanent_uuid: "7feabb2b6b524440bcd26b7158d241f2" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42967 } } peers { permanent_uuid: "ac54a7903d8e4405ab898e38f557aad1" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45671 } }
I20251024 20:01:25.268271 20426 ts_tablet_manager.cc:1434] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2: Time spent starting tablet: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:25.268337 20332 heartbeater.cc:499] Master 127.18.153.126:45119 was elected leader, sending a full tablet report...
W20251024 20:01:25.450809 20257 tablet.cc:2378] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 20:01:25.522303 20427 raft_consensus.cc:493] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 20:01:25.522420 20427 raft_consensus.cc:515] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42755 } } peers { permanent_uuid: "7feabb2b6b524440bcd26b7158d241f2" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42967 } } peers { permanent_uuid: "ac54a7903d8e4405ab898e38f557aad1" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45671 } }
I20251024 20:01:25.522691 20427 leader_election.cc:290] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b59e5f6e6bd4a8697ffac00418ffc4e (127.18.153.65:42755), 7feabb2b6b524440bcd26b7158d241f2 (127.18.153.66:42967)
I20251024 20:01:25.525403 20230 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35233874510d490b9db81f13f82965f3" candidate_uuid: "ac54a7903d8e4405ab898e38f557aad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e" is_pre_election: true
I20251024 20:01:25.525403 20306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35233874510d490b9db81f13f82965f3" candidate_uuid: "ac54a7903d8e4405ab898e38f557aad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7feabb2b6b524440bcd26b7158d241f2" is_pre_election: true
I20251024 20:01:25.525557 20230 raft_consensus.cc:2468] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac54a7903d8e4405ab898e38f557aad1 in term 0.
I20251024 20:01:25.525557 20306 raft_consensus.cc:2468] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac54a7903d8e4405ab898e38f557aad1 in term 0.
I20251024 20:01:25.525759 20345 leader_election.cc:304] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7feabb2b6b524440bcd26b7158d241f2, ac54a7903d8e4405ab898e38f557aad1; no voters: 
I20251024 20:01:25.525887 20427 raft_consensus.cc:2804] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 20:01:25.525923 20427 raft_consensus.cc:493] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 20:01:25.525952 20427 raft_consensus.cc:3060] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:25.526618 20427 raft_consensus.cc:515] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42755 } } peers { permanent_uuid: "7feabb2b6b524440bcd26b7158d241f2" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42967 } } peers { permanent_uuid: "ac54a7903d8e4405ab898e38f557aad1" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45671 } }
I20251024 20:01:25.526755 20427 leader_election.cc:290] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [CANDIDATE]: Term 1 election: Requested vote from peers 9b59e5f6e6bd4a8697ffac00418ffc4e (127.18.153.65:42755), 7feabb2b6b524440bcd26b7158d241f2 (127.18.153.66:42967)
I20251024 20:01:25.526878 20230 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35233874510d490b9db81f13f82965f3" candidate_uuid: "ac54a7903d8e4405ab898e38f557aad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e"
I20251024 20:01:25.526916 20306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35233874510d490b9db81f13f82965f3" candidate_uuid: "ac54a7903d8e4405ab898e38f557aad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7feabb2b6b524440bcd26b7158d241f2"
I20251024 20:01:25.526973 20230 raft_consensus.cc:3060] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:25.526983 20306 raft_consensus.cc:3060] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:25.527766 20230 raft_consensus.cc:2468] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac54a7903d8e4405ab898e38f557aad1 in term 1.
I20251024 20:01:25.527766 20306 raft_consensus.cc:2468] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac54a7903d8e4405ab898e38f557aad1 in term 1.
I20251024 20:01:25.527984 20346 leader_election.cc:304] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b59e5f6e6bd4a8697ffac00418ffc4e, ac54a7903d8e4405ab898e38f557aad1; no voters: 
I20251024 20:01:25.528108 20427 raft_consensus.cc:2804] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:01:25.528260 20427 raft_consensus.cc:697] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [term 1 LEADER]: Becoming Leader. State: Replica: ac54a7903d8e4405ab898e38f557aad1, State: Running, Role: LEADER
I20251024 20:01:25.528314 20427 consensus_queue.cc:237] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42755 } } peers { permanent_uuid: "7feabb2b6b524440bcd26b7158d241f2" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42967 } } peers { permanent_uuid: "ac54a7903d8e4405ab898e38f557aad1" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45671 } }
I20251024 20:01:25.528882 20127 catalog_manager.cc:5649] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac54a7903d8e4405ab898e38f557aad1 (127.18.153.67). New cstate: current_term: 1 leader_uuid: "ac54a7903d8e4405ab898e38f557aad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7feabb2b6b524440bcd26b7158d241f2" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac54a7903d8e4405ab898e38f557aad1" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45671 } health_report { overall_health: HEALTHY } } }
I20251024 20:01:26.030072 20435 consensus_queue.cc:1048] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7feabb2b6b524440bcd26b7158d241f2" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:01:26.031661 20438 consensus_queue.cc:1048] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b59e5f6e6bd4a8697ffac00418ffc4e" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:01:26.571933 19045 tablet_replica.cc:333] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e: stopping tablet replica
I20251024 20:01:26.572049 19045 raft_consensus.cc:2243] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:26.572100 19045 raft_consensus.cc:2272] T 35233874510d490b9db81f13f82965f3 P 9b59e5f6e6bd4a8697ffac00418ffc4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:26.934262 20346 consensus_queue.cc:579] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [LEADER]: The tablet replica hosted on peer 9b59e5f6e6bd4a8697ffac00418ffc4e has failed
W20251024 20:01:26.934360 20346 consensus_peers.cc:597] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 -> Peer 9b59e5f6e6bd4a8697ffac00418ffc4e (127.18.153.65:42755): Couldn't send request to peer 9b59e5f6e6bd4a8697ffac00418ffc4e. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20251024 20:01:27.085510 19045 tablet_server.cc:178] TabletServer@127.18.153.65:0 shutting down...
I20251024 20:01:27.088490 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:27.100226 19045 tablet_server.cc:195] TabletServer@127.18.153.65:0 shutdown complete.
I20251024 20:01:27.101152 19045 tablet_server.cc:178] TabletServer@127.18.153.66:0 shutting down...
I20251024 20:01:27.103744 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:27.103873 19045 tablet_replica.cc:333] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2: stopping tablet replica
I20251024 20:01:27.103941 19045 raft_consensus.cc:2243] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:27.103998 19045 raft_consensus.cc:2272] T 35233874510d490b9db81f13f82965f3 P 7feabb2b6b524440bcd26b7158d241f2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 20:01:27.105793 20345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.153.66:42967: connect: Connection refused (error 111)
W20251024 20:01:27.106176 20345 consensus_peers.cc:597] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 -> Peer 7feabb2b6b524440bcd26b7158d241f2 (127.18.153.66:42967): Couldn't send request to peer 7feabb2b6b524440bcd26b7158d241f2. Status: Network error: Client connection negotiation failed: client connection to 127.18.153.66:42967: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251024 20:01:27.115813 19045 tablet_server.cc:195] TabletServer@127.18.153.66:0 shutdown complete.
I20251024 20:01:27.116648 19045 tablet_server.cc:178] TabletServer@127.18.153.67:0 shutting down...
I20251024 20:01:27.119112 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:27.119235 19045 tablet_replica.cc:333] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1: stopping tablet replica
I20251024 20:01:27.119287 19045 raft_consensus.cc:2243] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:01:27.119400 19045 raft_consensus.cc:2272] T 35233874510d490b9db81f13f82965f3 P ac54a7903d8e4405ab898e38f557aad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:27.132164 19045 tablet_server.cc:195] TabletServer@127.18.153.67:0 shutdown complete.
I20251024 20:01:27.132985 19045 master.cc:561] Master@127.18.153.126:45119 shutting down...
I20251024 20:01:27.135181 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:01:27.135254 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:27.135277 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3fce915e481c4f34a63d518c0286e307: stopping tablet replica
I20251024 20:01:27.146891 19045 master.cc:583] Master@127.18.153.126:45119 shutdown complete.
[       OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (2023 ms)
[ RUN      ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20251024 20:01:27.151686 19045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.153.126:32845
I20251024 20:01:27.151854 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:27.152961 20452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:27.153095 20455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:27.153035 20453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:27.153314 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:27.153409 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:27.153437 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:27.153450 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336087153450 us; error 0 us; skew 500 ppm
I20251024 20:01:27.154013 19045 webserver.cc:492] Webserver started at http://127.18.153.126:39461/ using document root <none> and password file <none>
I20251024 20:01:27.154104 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:27.154150 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:27.154186 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:27.154435 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1761336053687453-19045-0/minicluster-data/master-0-root/instance:
uuid: "fbf238d6e7d94585972ae8848847a327"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.155278 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:27.155833 20460 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.155942 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.155989 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1761336053687453-19045-0/minicluster-data/master-0-root
uuid: "fbf238d6e7d94585972ae8848847a327"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.156038 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1761336053687453-19045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1761336053687453-19045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1761336053687453-19045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:27.176069 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:27.176256 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:27.178855 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.126:32845
I20251024 20:01:27.180774 20512 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.126:32845 every 8 connection(s)
I20251024 20:01:27.180928 20513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:27.182053 20513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327: Bootstrap starting.
I20251024 20:01:27.182361 20513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:27.182878 20513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327: No bootstrap required, opened a new log
I20251024 20:01:27.183010 20513 raft_consensus.cc:359] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf238d6e7d94585972ae8848847a327" member_type: VOTER }
I20251024 20:01:27.183063 20513 raft_consensus.cc:385] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:27.183084 20513 raft_consensus.cc:740] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf238d6e7d94585972ae8848847a327, State: Initialized, Role: FOLLOWER
I20251024 20:01:27.183140 20513 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf238d6e7d94585972ae8848847a327" member_type: VOTER }
I20251024 20:01:27.183182 20513 raft_consensus.cc:399] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:01:27.183209 20513 raft_consensus.cc:493] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:01:27.183236 20513 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:27.183753 20513 raft_consensus.cc:515] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf238d6e7d94585972ae8848847a327" member_type: VOTER }
I20251024 20:01:27.183815 20513 leader_election.cc:304] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbf238d6e7d94585972ae8848847a327; no voters: 
I20251024 20:01:27.183928 20513 leader_election.cc:290] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:01:27.183991 20516 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:01:27.184075 20513 sys_catalog.cc:565] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:01:27.184161 20516 raft_consensus.cc:697] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [term 1 LEADER]: Becoming Leader. State: Replica: fbf238d6e7d94585972ae8848847a327, State: Running, Role: LEADER
I20251024 20:01:27.184226 20516 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf238d6e7d94585972ae8848847a327" member_type: VOTER }
I20251024 20:01:27.184449 20517 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbf238d6e7d94585972ae8848847a327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf238d6e7d94585972ae8848847a327" member_type: VOTER } }
I20251024 20:01:27.184466 20518 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbf238d6e7d94585972ae8848847a327. Latest consensus state: current_term: 1 leader_uuid: "fbf238d6e7d94585972ae8848847a327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf238d6e7d94585972ae8848847a327" member_type: VOTER } }
I20251024 20:01:27.184526 20517 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:27.184551 20518 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:27.184904 20521 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:01:27.185053 20521 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:01:27.185504 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251024 20:01:27.185622 20521 catalog_manager.cc:1357] Generated new cluster ID: b214808df89146279d5812fbfd9745ef
I20251024 20:01:27.185670 20521 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:01:27.214398 20521 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 20:01:27.214802 20521 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:01:27.229578 20521 catalog_manager.cc:6022] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327: Generated new TSK 0
I20251024 20:01:27.229681 20521 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:01:27.249493 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:27.250833 20534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:27.250937 20535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:27.250979 19045 server_base.cc:1047] running on GCE node
W20251024 20:01:27.250869 20537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:27.251183 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:27.251221 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:27.251235 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336087251234 us; error 0 us; skew 500 ppm
I20251024 20:01:27.251891 19045 webserver.cc:492] Webserver started at http://127.18.153.65:38165/ using document root <none> and password file <none>
I20251024 20:01:27.251986 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:27.252031 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:27.252076 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:27.252347 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1761336053687453-19045-0/minicluster-data/ts-0-root/instance:
uuid: "953270cf9a564da6af7b41d4bf79dce3"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.253209 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:27.253652 20542 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.253780 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.253829 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "953270cf9a564da6af7b41d4bf79dce3"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.253875 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:27.275475 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:27.275683 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:27.275822 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:27.276023 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:27.276055 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.276079 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:27.276108 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.278839 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:40593
I20251024 20:01:27.278925 20605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:40593 every 8 connection(s)
I20251024 20:01:27.282191 20606 heartbeater.cc:344] Connected to a master server at 127.18.153.126:32845
I20251024 20:01:27.282258 20606 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:27.282357 20606 heartbeater.cc:507] Master 127.18.153.126:32845 requested a full tablet report, sending...
I20251024 20:01:27.282579 20477 ts_manager.cc:194] Registered new tserver with Master: 953270cf9a564da6af7b41d4bf79dce3 (127.18.153.65:40593)
I20251024 20:01:27.283133 20477 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34430
I20251024 20:01:27.283468 19045 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004489339s
I20251024 20:01:27.287921 19045 tablet_server.cc:178] TabletServer@127.18.153.65:0 shutting down...
I20251024 20:01:27.290239 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:27.301622 19045 tablet_server.cc:195] TabletServer@127.18.153.65:0 shutdown complete.
I20251024 20:01:27.302582 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251024 20:01:27.302775 19045 tablet_server.cc:93] deduplicated master addresses: 127.18.153.126:32845
W20251024 20:01:27.303821 20616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:27.303889 20617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:27.303966 19045 server_base.cc:1047] running on GCE node
W20251024 20:01:27.304034 20619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:27.304157 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:27.304190 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:27.304203 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336087304203 us; error 0 us; skew 500 ppm
I20251024 20:01:27.304785 19045 webserver.cc:492] Webserver started at http://127.18.153.65:38165/ using document root <none> and password file <none>
I20251024 20:01:27.304857 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:27.304899 19045 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251024 20:01:27.305465 19045 fs_manager.cc:714] Time spent opening directory manager: real 0.000s	user 0.001s	sys 0.000s
I20251024 20:01:27.305831 20624 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.305964 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.306003 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "953270cf9a564da6af7b41d4bf79dce3"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.306057 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:27.325057 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:27.325291 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:27.325464 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:27.325659 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:27.325686 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.325722 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:27.325749 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.328622 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:40593
I20251024 20:01:27.328670 20687 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:40593 every 8 connection(s)
I20251024 20:01:27.329653 19045 tablet_server.cc:178] TabletServer@127.18.153.65:40593 shutting down...
I20251024 20:01:27.332861 20688 heartbeater.cc:344] Connected to a master server at 127.18.153.126:32845
I20251024 20:01:27.332930 20688 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:27.333034 20688 heartbeater.cc:507] Master 127.18.153.126:32845 requested a full tablet report, sending...
I20251024 20:01:27.333287 20477 ts_manager.cc:194] Re-registered known tserver with Master: 953270cf9a564da6af7b41d4bf79dce3 (127.18.153.65:40593)
I20251024 20:01:27.333618 20477 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34444
I20251024 20:01:27.334069 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:27.345641 19045 tablet_server.cc:195] TabletServer@127.18.153.65:40593 shutdown complete.
I20251024 20:01:27.346623 19045 master.cc:561] Master@127.18.153.126:32845 shutting down...
I20251024 20:01:27.348959 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:01:27.349042 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:27.349128 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P fbf238d6e7d94585972ae8848847a327: stopping tablet replica
I20251024 20:01:27.361081 19045 master.cc:583] Master@127.18.153.126:32845 shutdown complete.
[       OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (212 ms)
[ RUN      ] TsTabletManagerITest.TestTableStats
I20251024 20:01:27.364559 19045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.153.126:38517,127.18.153.125:38773,127.18.153.124:38423
I20251024 20:01:27.364737 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:27.365854 20695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:27.365873 20698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:27.365952 20696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:27.366050 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:27.366227 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:27.366269 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:27.366282 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336087366283 us; error 0 us; skew 500 ppm
I20251024 20:01:27.366858 19045 webserver.cc:492] Webserver started at http://127.18.153.126:35241/ using document root <none> and password file <none>
I20251024 20:01:27.366941 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:27.366984 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:27.367039 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:27.367307 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-0-root/instance:
uuid: "a339d74b39234a11a469995395989e7a"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.368223 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:27.368655 20703 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.368808 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.368860 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-0-root
uuid: "a339d74b39234a11a469995395989e7a"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.368901 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:27.394814 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:27.395004 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:27.397625 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.126:38517
I20251024 20:01:27.398343 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251024 20:01:27.399147 20755 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.126:38517 every 8 connection(s)
I20251024 20:01:27.399453 20756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 20:01:27.400107 20758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:27.400177 19045 server_base.cc:1047] running on GCE node
W20251024 20:01:27.400277 20759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:27.400116 20761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:27.400420 20756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } has no permanent_uuid. Determining permanent_uuid...
I20251024 20:01:27.400475 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:27.400523 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:27.400542 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336087400542 us; error 0 us; skew 500 ppm
I20251024 20:01:27.401199 19045 webserver.cc:492] Webserver started at http://127.18.153.125:40191/ using document root <none> and password file <none>
I20251024 20:01:27.401293 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:27.401337 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:27.401396 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:27.401674 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-1-root/instance:
uuid: "df8fdb366ca94c638b9886b02d552ed7"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.402789 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:27.403345 20768 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.403697 19045 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
I20251024 20:01:27.403744 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-1-root
uuid: "df8fdb366ca94c638b9886b02d552ed7"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.403800 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:27.404453 20756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } has no permanent_uuid. Determining permanent_uuid...
W20251024 20:01:27.405035 20756 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.153.125:38773: Network error: Client connection negotiation failed: client connection to 127.18.153.125:38773: connect: Connection refused (error 111)
I20251024 20:01:27.415117 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:27.415403 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:27.417965 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.125:38773
I20251024 20:01:27.418006 20821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.125:38773 every 8 connection(s)
I20251024 20:01:27.418283 20822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:27.418293 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251024 20:01:27.419247 20822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } has no permanent_uuid. Determining permanent_uuid...
W20251024 20:01:27.419486 20825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:27.419549 20824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:27.419720 20827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:27.419904 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:27.420065 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:27.420104 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:27.420121 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336087420122 us; error 0 us; skew 500 ppm
I20251024 20:01:27.420733 19045 webserver.cc:492] Webserver started at http://127.18.153.124:39553/ using document root <none> and password file <none>
I20251024 20:01:27.420812 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:27.420876 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:27.420934 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:27.421240 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-2-root/instance:
uuid: "496a3c55c0994110b7824e8e0ea4015a"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.422355 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:27.422820 20822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } has no permanent_uuid. Determining permanent_uuid...
I20251024 20:01:27.422930 20833 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.423069 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.001s	sys 0.000s
I20251024 20:01:27.423134 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-2-root
uuid: "496a3c55c0994110b7824e8e0ea4015a"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.423187 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:27.426077 20822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } has no permanent_uuid. Determining permanent_uuid...
W20251024 20:01:27.426712 20822 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.153.124:38423: Network error: Client connection negotiation failed: client connection to 127.18.153.124:38423: connect: Connection refused (error 111)
I20251024 20:01:27.428395 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:27.428597 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:27.431565 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.124:38423
I20251024 20:01:27.431612 20887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.124:38423 every 8 connection(s)
I20251024 20:01:27.431775 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251024 20:01:27.431869 20888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:27.432492 20888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } has no permanent_uuid. Determining permanent_uuid...
I20251024 20:01:27.435241 20888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } has no permanent_uuid. Determining permanent_uuid...
I20251024 20:01:27.437806 20888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } has no permanent_uuid. Determining permanent_uuid...
I20251024 20:01:27.441205 20888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a: Bootstrap starting.
I20251024 20:01:27.441205 20756 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } attempt: 1
I20251024 20:01:27.441663 20888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:27.442255 20888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a: No bootstrap required, opened a new log
I20251024 20:01:27.442448 20888 raft_consensus.cc:359] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:27.442525 20888 raft_consensus.cc:385] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:27.442548 20888 raft_consensus.cc:740] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 496a3c55c0994110b7824e8e0ea4015a, State: Initialized, Role: FOLLOWER
I20251024 20:01:27.442595 20888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:27.442775 20888 sys_catalog.cc:565] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:01:27.442781 20893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:27.442891 20893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: This master's current role is: FOLLOWER
W20251024 20:01:27.444252 20904 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251024 20:01:27.444362 20904 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251024 20:01:27.444840 20756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } has no permanent_uuid. Determining permanent_uuid...
I20251024 20:01:27.447803 20756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a: Bootstrap starting.
I20251024 20:01:27.448194 20756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:27.448729 20756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a: No bootstrap required, opened a new log
I20251024 20:01:27.448875 20756 raft_consensus.cc:359] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:27.448953 20756 raft_consensus.cc:385] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:27.448971 20756 raft_consensus.cc:740] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a339d74b39234a11a469995395989e7a, State: Initialized, Role: FOLLOWER
I20251024 20:01:27.449008 20756 consensus_queue.cc:260] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:27.449186 20756 sys_catalog.cc:565] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:01:27.449193 20906 sys_catalog.cc:455] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:27.449283 20906 sys_catalog.cc:458] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:27.450321 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20251024 20:01:27.450464 20917 catalog_manager.cc:1568] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251024 20:01:27.450497 20917 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251024 20:01:27.451170 20906 raft_consensus.cc:493] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 20:01:27.451211 20906 raft_consensus.cc:515] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:27.451344 20906 leader_election.cc:290] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773), 496a3c55c0994110b7824e8e0ea4015a (127.18.153.124:38423)
I20251024 20:01:27.451575 20862 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a339d74b39234a11a469995395989e7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "496a3c55c0994110b7824e8e0ea4015a" is_pre_election: true
I20251024 20:01:27.451651 20862 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a339d74b39234a11a469995395989e7a in term 0.
I20251024 20:01:27.451846 20706 leader_election.cc:304] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 496a3c55c0994110b7824e8e0ea4015a, a339d74b39234a11a469995395989e7a; no voters: 
I20251024 20:01:27.451934 20906 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 20:01:27.451987 20906 raft_consensus.cc:493] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 20:01:27.452011 20906 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:27.452848 20906 raft_consensus.cc:515] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:27.452984 20906 leader_election.cc:290] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [CANDIDATE]: Term 1 election: Requested vote from peers df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773), 496a3c55c0994110b7824e8e0ea4015a (127.18.153.124:38423)
I20251024 20:01:27.453197 20862 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a339d74b39234a11a469995395989e7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "496a3c55c0994110b7824e8e0ea4015a"
I20251024 20:01:27.453254 20862 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:27.453997 20862 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a339d74b39234a11a469995395989e7a in term 1.
I20251024 20:01:27.454176 20706 leader_election.cc:304] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 496a3c55c0994110b7824e8e0ea4015a, a339d74b39234a11a469995395989e7a; no voters: 
I20251024 20:01:27.454249 20906 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:01:27.454366 20906 raft_consensus.cc:697] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 1 LEADER]: Becoming Leader. State: Replica: a339d74b39234a11a469995395989e7a, State: Running, Role: LEADER
I20251024 20:01:27.454418 20906 consensus_queue.cc:237] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:27.454715 20918 sys_catalog.cc:455] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a339d74b39234a11a469995395989e7a. Latest consensus state: current_term: 1 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:27.454818 20918 sys_catalog.cc:458] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:27.455006 20920 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:01:27.455173 20920 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:01:27.455826 20862 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 1 FOLLOWER]: Refusing update from remote peer a339d74b39234a11a469995395989e7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 20:01:27.456032 20918 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:01:27.456707 20893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a339d74b39234a11a469995395989e7a. Latest consensus state: current_term: 1 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:27.456784 20893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:27.457252 20906 sys_catalog.cc:455] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:27.457340 20906 sys_catalog.cc:458] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:27.457412 20921 mvcc.cc:204] Tried to move back new op lower bound from 7214432614217760768 to 7214432614213832704. Current Snapshot: MvccSnapshot[applied={T|T < 7214432614217760768 or (T in {7214432614217760768})}]
I20251024 20:01:27.457468 20893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:27.457559 20893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:27.457414 20920 catalog_manager.cc:1357] Generated new cluster ID: bbb04eb8879d4aa8a0e7ff573f23a582
I20251024 20:01:27.457621 20920 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:01:27.465332 20920 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 20:01:27.465945 20920 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:01:27.473584 20920 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a: Generated new TSK 0
I20251024 20:01:27.473719 20920 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:01:27.476862 20822 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } attempt: 1
I20251024 20:01:27.481256 20822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: Bootstrap starting.
I20251024 20:01:27.481714 20822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:27.482504 20822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: No bootstrap required, opened a new log
I20251024 20:01:27.482698 20822 raft_consensus.cc:359] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:27.482761 20822 raft_consensus.cc:385] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:27.482786 20822 raft_consensus.cc:740] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8fdb366ca94c638b9886b02d552ed7, State: Initialized, Role: FOLLOWER
I20251024 20:01:27.482843 20822 consensus_queue.cc:260] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:27.483045 20822 sys_catalog.cc:565] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:01:27.483098 20794 raft_consensus.cc:3060] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:27.483076 20927 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:27.483186 20927 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:27.483850 20794 raft_consensus.cc:1275] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 1 FOLLOWER]: Refusing update from remote peer a339d74b39234a11a469995395989e7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 20:01:27.483119 20796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a339d74b39234a11a469995395989e7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8fdb366ca94c638b9886b02d552ed7" is_pre_election: true
I20251024 20:01:27.484033 20796 raft_consensus.cc:2468] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a339d74b39234a11a469995395989e7a in term 1.
I20251024 20:01:27.484127 20906 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:01:27.484536 20795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a339d74b39234a11a469995395989e7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8fdb366ca94c638b9886b02d552ed7"
I20251024 20:01:27.485076 20927 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a339d74b39234a11a469995395989e7a. Latest consensus state: current_term: 1 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:27.485139 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20251024 20:01:27.485121 20940 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:27.485162 20927 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:27.485208 20940 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:27.485397 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251024 20:01:27.485447 20944 catalog_manager.cc:1269] Loaded cluster ID: bbb04eb8879d4aa8a0e7ff573f23a582
I20251024 20:01:27.485489 20944 catalog_manager.cc:1562] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: loading cluster ID for follower catalog manager: success
I20251024 20:01:27.486411 20944 catalog_manager.cc:1584] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: acquiring CA information for follower catalog manager: success
W20251024 20:01:27.486562 20948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:27.486653 20944 catalog_manager.cc:1612] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251024 20:01:27.486688 20946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:27.486884 20945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:27.486907 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:27.487073 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:27.487102 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:27.487123 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336087487123 us; error 0 us; skew 500 ppm
I20251024 20:01:27.487799 19045 webserver.cc:492] Webserver started at http://127.18.153.65:40607/ using document root <none> and password file <none>
I20251024 20:01:27.487885 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:27.487931 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:27.487983 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:27.488240 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-0-root/instance:
uuid: "300038e5d9694754b6855f62e5e5e679"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.489073 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:27.489512 20953 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.489660 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.489702 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "300038e5d9694754b6855f62e5e5e679"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.489753 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:27.507126 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:27.507455 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:27.507638 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:27.507859 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:27.507889 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.507926 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:27.507954 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.511189 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:33453
I20251024 20:01:27.511289 21016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:33453 every 8 connection(s)
I20251024 20:01:27.511749 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:27.513254 21024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:27.513744 21027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:27.513942 21025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:27.514031 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:27.514367 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:27.514420 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:27.515072 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336087515069 us; error 0 us; skew 500 ppm
I20251024 20:01:27.515779 19045 webserver.cc:492] Webserver started at http://127.18.153.66:34415/ using document root <none> and password file <none>
I20251024 20:01:27.515870 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:27.515913 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:27.515957 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:27.516110 21018 heartbeater.cc:344] Connected to a master server at 127.18.153.126:38517
I20251024 20:01:27.516166 21018 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:27.516280 21018 heartbeater.cc:507] Master 127.18.153.126:38517 requested a full tablet report, sending...
I20251024 20:01:27.516292 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-1-root/instance:
uuid: "e0594f5433ae41248b6004bf70361182"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.516527 20720 ts_manager.cc:194] Registered new tserver with Master: 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453)
I20251024 20:01:27.517202 20720 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49264
I20251024 20:01:27.517519 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.000s
I20251024 20:01:27.522845 21032 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.522943 21019 heartbeater.cc:344] Connected to a master server at 127.18.153.125:38773
I20251024 20:01:27.523259 19045 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.000s	sys 0.000s
I20251024 20:01:27.523286 21019 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:27.523296 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-1-root
uuid: "e0594f5433ae41248b6004bf70361182"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.523340 21019 heartbeater.cc:507] Master 127.18.153.125:38773 requested a full tablet report, sending...
I20251024 20:01:27.523352 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:27.523650 20786 ts_manager.cc:194] Registered new tserver with Master: 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453)
I20251024 20:01:27.525063 21017 heartbeater.cc:344] Connected to a master server at 127.18.153.124:38423
I20251024 20:01:27.525122 21017 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:27.525167 21017 heartbeater.cc:507] Master 127.18.153.124:38423 requested a full tablet report, sending...
I20251024 20:01:27.525485 20852 ts_manager.cc:194] Registered new tserver with Master: 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453)
I20251024 20:01:27.551120 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:27.551345 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:27.551565 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:27.551786 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:27.551815 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.551836 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:27.551864 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.555243 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.66:38339
I20251024 20:01:27.555351 21095 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.66:38339 every 8 connection(s)
I20251024 20:01:27.555801 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:27.557127 21104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:27.557512 21106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:27.557924 19045 server_base.cc:1047] running on GCE node
W20251024 20:01:27.558032 21103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:27.558693 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:27.558739 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:27.558764 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336087558764 us; error 0 us; skew 500 ppm
I20251024 20:01:27.559500 19045 webserver.cc:492] Webserver started at http://127.18.153.67:34091/ using document root <none> and password file <none>
I20251024 20:01:27.559589 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:27.559638 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:27.559679 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:27.559713 21097 heartbeater.cc:344] Connected to a master server at 127.18.153.126:38517
I20251024 20:01:27.559806 21097 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:27.559913 21097 heartbeater.cc:507] Master 127.18.153.126:38517 requested a full tablet report, sending...
I20251024 20:01:27.560041 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-2-root/instance:
uuid: "8b22fe88c0914cdeb565ee21f8b66cd4"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.560171 20720 ts_manager.cc:194] Registered new tserver with Master: e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339)
I20251024 20:01:27.560480 21096 heartbeater.cc:344] Connected to a master server at 127.18.153.124:38423
I20251024 20:01:27.560527 21096 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:27.560604 21096 heartbeater.cc:507] Master 127.18.153.124:38423 requested a full tablet report, sending...
I20251024 20:01:27.560652 20720 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49266
I20251024 20:01:27.560487 21098 heartbeater.cc:344] Connected to a master server at 127.18.153.125:38773
I20251024 20:01:27.560720 21098 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:27.560937 21098 heartbeater.cc:507] Master 127.18.153.125:38773 requested a full tablet report, sending...
I20251024 20:01:27.560993 20852 ts_manager.cc:194] Registered new tserver with Master: e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339)
I20251024 20:01:27.561184 20786 ts_manager.cc:194] Registered new tserver with Master: e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339)
I20251024 20:01:27.561210 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.002s
I20251024 20:01:27.561698 21111 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.561803 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.561851 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-2-root
uuid: "8b22fe88c0914cdeb565ee21f8b66cd4"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:27.561895 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:27.570165 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:27.570444 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:27.570611 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:27.570819 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:27.570847 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.570869 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:27.570899 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.574229 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.67:45221
I20251024 20:01:27.574504 21174 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.67:45221 every 8 connection(s)
I20251024 20:01:27.578171 21177 heartbeater.cc:344] Connected to a master server at 127.18.153.125:38773
I20251024 20:01:27.578171 21176 heartbeater.cc:344] Connected to a master server at 127.18.153.126:38517
I20251024 20:01:27.578263 21177 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:27.578351 21176 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:27.578444 21176 heartbeater.cc:507] Master 127.18.153.126:38517 requested a full tablet report, sending...
I20251024 20:01:27.578454 21177 heartbeater.cc:507] Master 127.18.153.125:38773 requested a full tablet report, sending...
I20251024 20:01:27.578536 21175 heartbeater.cc:344] Connected to a master server at 127.18.153.124:38423
I20251024 20:01:27.578605 21175 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:27.578703 20720 ts_manager.cc:194] Registered new tserver with Master: 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:27.578720 20786 ts_manager.cc:194] Registered new tserver with Master: 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:27.578693 21175 heartbeater.cc:507] Master 127.18.153.124:38423 requested a full tablet report, sending...
I20251024 20:01:27.578980 20852 ts_manager.cc:194] Registered new tserver with Master: 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:27.579147 20720 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49276
I20251024 20:01:27.579617 19045 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005110203s
I20251024 20:01:27.586005 20720 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49290:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20251024 20:01:27.586169 20720 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 20:01:27.586736 20906 sys_catalog.cc:455] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:27.586813 20906 sys_catalog.cc:458] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:27.592635 21139 tablet_service.cc:1505] Processing CreateTablet for tablet a4e6dc18db1e49208af2fa5dfaae96a6 (DEFAULT_TABLE table=test-table [id=ef884b5c6c8b4ed6bc23772e0adeb300]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251024 20:01:27.592635 21060 tablet_service.cc:1505] Processing CreateTablet for tablet a4e6dc18db1e49208af2fa5dfaae96a6 (DEFAULT_TABLE table=test-table [id=ef884b5c6c8b4ed6bc23772e0adeb300]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251024 20:01:27.592764 21137 tablet_service.cc:1505] Processing CreateTablet for tablet 784c0490ffca4ccaa5cece7bae6064c2 (DEFAULT_TABLE table=test-table [id=ef884b5c6c8b4ed6bc23772e0adeb300]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20251024 20:01:27.592828 21139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e6dc18db1e49208af2fa5dfaae96a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:27.592859 21060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e6dc18db1e49208af2fa5dfaae96a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:27.592885 21058 tablet_service.cc:1505] Processing CreateTablet for tablet 784c0490ffca4ccaa5cece7bae6064c2 (DEFAULT_TABLE table=test-table [id=ef884b5c6c8b4ed6bc23772e0adeb300]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20251024 20:01:27.592984 21058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 784c0490ffca4ccaa5cece7bae6064c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:27.592717 21059 tablet_service.cc:1505] Processing CreateTablet for tablet 2dbef0e1505045468e973459c151f622 (DEFAULT_TABLE table=test-table [id=ef884b5c6c8b4ed6bc23772e0adeb300]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251024 20:01:27.593125 21059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dbef0e1505045468e973459c151f622. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:27.592676 21138 tablet_service.cc:1505] Processing CreateTablet for tablet 2dbef0e1505045468e973459c151f622 (DEFAULT_TABLE table=test-table [id=ef884b5c6c8b4ed6bc23772e0adeb300]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251024 20:01:27.593218 21138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dbef0e1505045468e973459c151f622. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:27.593273 21137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 784c0490ffca4ccaa5cece7bae6064c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:27.593578 20981 tablet_service.cc:1505] Processing CreateTablet for tablet a4e6dc18db1e49208af2fa5dfaae96a6 (DEFAULT_TABLE table=test-table [id=ef884b5c6c8b4ed6bc23772e0adeb300]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251024 20:01:27.593595 20980 tablet_service.cc:1505] Processing CreateTablet for tablet 2dbef0e1505045468e973459c151f622 (DEFAULT_TABLE table=test-table [id=ef884b5c6c8b4ed6bc23772e0adeb300]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251024 20:01:27.593595 20979 tablet_service.cc:1505] Processing CreateTablet for tablet 784c0490ffca4ccaa5cece7bae6064c2 (DEFAULT_TABLE table=test-table [id=ef884b5c6c8b4ed6bc23772e0adeb300]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20251024 20:01:27.593706 20981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e6dc18db1e49208af2fa5dfaae96a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:27.593748 20980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dbef0e1505045468e973459c151f622. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:27.593906 20979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 784c0490ffca4ccaa5cece7bae6064c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:27.617873 21197 tablet_bootstrap.cc:492] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4: Bootstrap starting.
I20251024 20:01:27.618398 21197 tablet_bootstrap.cc:654] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:27.618872 21198 tablet_bootstrap.cc:492] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182: Bootstrap starting.
I20251024 20:01:27.619218 21197 tablet_bootstrap.cc:492] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4: No bootstrap required, opened a new log
I20251024 20:01:27.619279 21197 ts_tablet_manager.cc:1403] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:27.619488 21197 raft_consensus.cc:359] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.619570 21197 raft_consensus.cc:385] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:27.619601 21197 raft_consensus.cc:740] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b22fe88c0914cdeb565ee21f8b66cd4, State: Initialized, Role: FOLLOWER
I20251024 20:01:27.619670 21197 consensus_queue.cc:260] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.619858 21197 ts_tablet_manager.cc:1434] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:27.619939 21198 tablet_bootstrap.cc:654] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:27.619998 21176 heartbeater.cc:499] Master 127.18.153.126:38517 was elected leader, sending a full tablet report...
I20251024 20:01:27.620358 21196 tablet_bootstrap.cc:492] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679: Bootstrap starting.
I20251024 20:01:27.620960 21197 tablet_bootstrap.cc:492] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4: Bootstrap starting.
I20251024 20:01:27.621024 21198 tablet_bootstrap.cc:492] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182: No bootstrap required, opened a new log
I20251024 20:01:27.621084 21198 ts_tablet_manager.cc:1403] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I20251024 20:01:27.621187 21196 tablet_bootstrap.cc:654] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:27.621264 21198 raft_consensus.cc:359] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.621351 21198 raft_consensus.cc:385] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:27.621377 21198 raft_consensus.cc:740] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0594f5433ae41248b6004bf70361182, State: Initialized, Role: FOLLOWER
I20251024 20:01:27.621431 21197 tablet_bootstrap.cc:654] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:27.621439 21198 consensus_queue.cc:260] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.621677 21198 ts_tablet_manager.cc:1434] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:27.621747 21097 heartbeater.cc:499] Master 127.18.153.126:38517 was elected leader, sending a full tablet report...
I20251024 20:01:27.621762 21198 tablet_bootstrap.cc:492] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182: Bootstrap starting.
I20251024 20:01:27.622084 21196 tablet_bootstrap.cc:492] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679: No bootstrap required, opened a new log
I20251024 20:01:27.622123 21196 ts_tablet_manager.cc:1403] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I20251024 20:01:27.622311 21196 raft_consensus.cc:359] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.622406 21196 raft_consensus.cc:385] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:27.622431 21196 raft_consensus.cc:740] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 300038e5d9694754b6855f62e5e5e679, State: Initialized, Role: FOLLOWER
I20251024 20:01:27.622502 21196 consensus_queue.cc:260] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.622610 21198 tablet_bootstrap.cc:654] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:27.622713 21196 ts_tablet_manager.cc:1434] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:27.622812 21196 tablet_bootstrap.cc:492] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679: Bootstrap starting.
I20251024 20:01:27.622833 21018 heartbeater.cc:499] Master 127.18.153.126:38517 was elected leader, sending a full tablet report...
I20251024 20:01:27.623282 21196 tablet_bootstrap.cc:654] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:27.623437 21197 tablet_bootstrap.cc:492] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4: No bootstrap required, opened a new log
I20251024 20:01:27.623483 21197 ts_tablet_manager.cc:1403] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.001s
I20251024 20:01:27.623673 21197 raft_consensus.cc:359] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.623741 21197 raft_consensus.cc:385] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:27.623764 21197 raft_consensus.cc:740] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b22fe88c0914cdeb565ee21f8b66cd4, State: Initialized, Role: FOLLOWER
I20251024 20:01:27.623808 21197 consensus_queue.cc:260] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.623895 21197 ts_tablet_manager.cc:1434] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4: Time spent starting tablet: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.623953 21197 tablet_bootstrap.cc:492] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4: Bootstrap starting.
I20251024 20:01:27.624365 21197 tablet_bootstrap.cc:654] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:27.624703 21198 tablet_bootstrap.cc:492] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182: No bootstrap required, opened a new log
I20251024 20:01:27.624768 21205 raft_consensus.cc:493] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 20:01:27.624747 21198 ts_tablet_manager.cc:1403] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
I20251024 20:01:27.624743 21196 tablet_bootstrap.cc:492] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679: No bootstrap required, opened a new log
I20251024 20:01:27.624890 21196 ts_tablet_manager.cc:1403] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I20251024 20:01:27.624975 21198 raft_consensus.cc:359] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.625051 21198 raft_consensus.cc:385] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:27.625082 21198 raft_consensus.cc:740] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0594f5433ae41248b6004bf70361182, State: Initialized, Role: FOLLOWER
I20251024 20:01:27.625069 21197 tablet_bootstrap.cc:492] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4: No bootstrap required, opened a new log
I20251024 20:01:27.625063 21196 raft_consensus.cc:359] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.624828 21205 raft_consensus.cc:515] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.625174 21197 ts_tablet_manager.cc:1403] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4: Time spent bootstrapping tablet: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:27.625195 21196 raft_consensus.cc:385] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:27.625281 21196 raft_consensus.cc:740] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 300038e5d9694754b6855f62e5e5e679, State: Initialized, Role: FOLLOWER
I20251024 20:01:27.625174 21198 consensus_queue.cc:260] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.625332 21196 consensus_queue.cc:260] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.625389 21205 leader_election.cc:290] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:27.625375 21197 raft_consensus.cc:359] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.625416 21198 ts_tablet_manager.cc:1434] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:27.625484 21196 ts_tablet_manager.cc:1434] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:27.625429 21197 raft_consensus.cc:385] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:27.625588 21197 raft_consensus.cc:740] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b22fe88c0914cdeb565ee21f8b66cd4, State: Initialized, Role: FOLLOWER
I20251024 20:01:27.625638 21197 consensus_queue.cc:260] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.625727 21197 ts_tablet_manager.cc:1434] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4: Time spent starting tablet: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.625540 21196 tablet_bootstrap.cc:492] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679: Bootstrap starting.
I20251024 20:01:27.625538 21198 tablet_bootstrap.cc:492] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182: Bootstrap starting.
I20251024 20:01:27.626663 21196 tablet_bootstrap.cc:654] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:27.626832 21198 tablet_bootstrap.cc:654] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:27.627517 21196 tablet_bootstrap.cc:492] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679: No bootstrap required, opened a new log
I20251024 20:01:27.627564 21196 ts_tablet_manager.cc:1403] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I20251024 20:01:27.627705 21196 raft_consensus.cc:359] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.627776 21196 raft_consensus.cc:385] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:27.627804 21196 raft_consensus.cc:740] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 300038e5d9694754b6855f62e5e5e679, State: Initialized, Role: FOLLOWER
I20251024 20:01:27.627857 21196 consensus_queue.cc:260] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.627954 21196 ts_tablet_manager.cc:1434] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679: Time spent starting tablet: real 0.000s	user 0.000s	sys 0.001s
I20251024 20:01:27.628147 21198 tablet_bootstrap.cc:492] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182: No bootstrap required, opened a new log
I20251024 20:01:27.628193 21198 ts_tablet_manager.cc:1403] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
I20251024 20:01:27.628367 21198 raft_consensus.cc:359] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.628427 21198 raft_consensus.cc:385] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:27.628453 21198 raft_consensus.cc:740] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0594f5433ae41248b6004bf70361182, State: Initialized, Role: FOLLOWER
I20251024 20:01:27.628499 21198 consensus_queue.cc:260] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.628580 21198 ts_tablet_manager.cc:1434] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182: Time spent starting tablet: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:27.630501 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182" is_pre_election: true
I20251024 20:01:27.630501 21149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" is_pre_election: true
I20251024 20:01:27.630609 21070 raft_consensus.cc:2468] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 300038e5d9694754b6855f62e5e5e679 in term 0.
I20251024 20:01:27.630609 21149 raft_consensus.cc:2468] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 300038e5d9694754b6855f62e5e5e679 in term 0.
I20251024 20:01:27.630789 20956 leader_election.cc:304] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 300038e5d9694754b6855f62e5e5e679, 8b22fe88c0914cdeb565ee21f8b66cd4; no voters: 
I20251024 20:01:27.630960 21209 raft_consensus.cc:2804] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 20:01:27.630999 21209 raft_consensus.cc:493] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 20:01:27.631014 21209 raft_consensus.cc:3060] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:27.631919 21209 raft_consensus.cc:515] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.632035 21209 leader_election.cc:290] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 1 election: Requested vote from peers e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:27.632186 21149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4"
I20251024 20:01:27.632246 21149 raft_consensus.cc:3060] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:27.632747 21149 raft_consensus.cc:2468] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 300038e5d9694754b6855f62e5e5e679 in term 1.
I20251024 20:01:27.632839 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182"
I20251024 20:01:27.632884 21070 raft_consensus.cc:3060] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:27.632897 20956 leader_election.cc:304] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 300038e5d9694754b6855f62e5e5e679, 8b22fe88c0914cdeb565ee21f8b66cd4; no voters: 
I20251024 20:01:27.632997 21209 raft_consensus.cc:2804] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:01:27.633041 21209 raft_consensus.cc:697] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 1 LEADER]: Becoming Leader. State: Replica: 300038e5d9694754b6855f62e5e5e679, State: Running, Role: LEADER
I20251024 20:01:27.633091 21209 consensus_queue.cc:237] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.633558 21070 raft_consensus.cc:2468] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 300038e5d9694754b6855f62e5e5e679 in term 1.
I20251024 20:01:27.633543 20720 catalog_manager.cc:5649] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 reported cstate change: term changed from 0 to 1, leader changed from <none> to 300038e5d9694754b6855f62e5e5e679 (127.18.153.65). New cstate: current_term: 1 leader_uuid: "300038e5d9694754b6855f62e5e5e679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } health_report { overall_health: UNKNOWN } } }
I20251024 20:01:27.635918 21200 raft_consensus.cc:493] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 20:01:27.635982 21200 raft_consensus.cc:515] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.636159 21200 leader_election.cc:290] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339), 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453)
I20251024 20:01:27.639254 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e6dc18db1e49208af2fa5dfaae96a6" candidate_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182" is_pre_election: true
I20251024 20:01:27.639338 21070 raft_consensus.cc:2468] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b22fe88c0914cdeb565ee21f8b66cd4 in term 0.
I20251024 20:01:27.639328 20991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e6dc18db1e49208af2fa5dfaae96a6" candidate_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "300038e5d9694754b6855f62e5e5e679" is_pre_election: true
I20251024 20:01:27.639484 20991 raft_consensus.cc:2468] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b22fe88c0914cdeb565ee21f8b66cd4 in term 0.
I20251024 20:01:27.639580 21112 leader_election.cc:304] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b22fe88c0914cdeb565ee21f8b66cd4, e0594f5433ae41248b6004bf70361182; no voters: 
I20251024 20:01:27.639650 21200 raft_consensus.cc:2804] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 20:01:27.639691 21200 raft_consensus.cc:493] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 20:01:27.639719 21200 raft_consensus.cc:3060] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:27.640298 21200 raft_consensus.cc:515] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.640424 21200 leader_election.cc:290] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [CANDIDATE]: Term 1 election: Requested vote from peers e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339), 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453)
I20251024 20:01:27.640535 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e6dc18db1e49208af2fa5dfaae96a6" candidate_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182"
I20251024 20:01:27.640587 21070 raft_consensus.cc:3060] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:27.640573 20991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e6dc18db1e49208af2fa5dfaae96a6" candidate_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "300038e5d9694754b6855f62e5e5e679"
I20251024 20:01:27.640652 20991 raft_consensus.cc:3060] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:27.641049 21070 raft_consensus.cc:2468] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b22fe88c0914cdeb565ee21f8b66cd4 in term 1.
I20251024 20:01:27.641199 21112 leader_election.cc:304] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b22fe88c0914cdeb565ee21f8b66cd4, e0594f5433ae41248b6004bf70361182; no voters: 
I20251024 20:01:27.641275 21200 raft_consensus.cc:2804] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:01:27.641330 20991 raft_consensus.cc:2468] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b22fe88c0914cdeb565ee21f8b66cd4 in term 1.
I20251024 20:01:27.641398 21200 raft_consensus.cc:697] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 LEADER]: Becoming Leader. State: Replica: 8b22fe88c0914cdeb565ee21f8b66cd4, State: Running, Role: LEADER
I20251024 20:01:27.641459 21200 consensus_queue.cc:237] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.641996 20720 catalog_manager.cc:5649] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67). New cstate: current_term: 1 leader_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } health_report { overall_health: HEALTHY } } }
I20251024 20:01:27.679948 21209 raft_consensus.cc:493] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 20:01:27.680073 21209 raft_consensus.cc:515] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.680279 21209 leader_election.cc:290] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:27.680505 21149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" is_pre_election: true
I20251024 20:01:27.680505 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182" is_pre_election: true
I20251024 20:01:27.680632 21149 raft_consensus.cc:2468] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 300038e5d9694754b6855f62e5e5e679 in term 0.
I20251024 20:01:27.680635 21070 raft_consensus.cc:2468] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 300038e5d9694754b6855f62e5e5e679 in term 0.
I20251024 20:01:27.680787 20956 leader_election.cc:304] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 300038e5d9694754b6855f62e5e5e679, 8b22fe88c0914cdeb565ee21f8b66cd4; no voters: 
I20251024 20:01:27.680892 21209 raft_consensus.cc:2804] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 20:01:27.680934 21209 raft_consensus.cc:493] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 20:01:27.680960 21209 raft_consensus.cc:3060] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:27.681574 21209 raft_consensus.cc:515] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.681690 21209 leader_election.cc:290] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 1 election: Requested vote from peers e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:27.681852 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182"
I20251024 20:01:27.681896 21149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4"
I20251024 20:01:27.681934 21070 raft_consensus.cc:3060] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:27.681941 21149 raft_consensus.cc:3060] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:27.682647 21149 raft_consensus.cc:2468] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 300038e5d9694754b6855f62e5e5e679 in term 1.
I20251024 20:01:27.682647 21070 raft_consensus.cc:2468] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 300038e5d9694754b6855f62e5e5e679 in term 1.
I20251024 20:01:27.682924 20956 leader_election.cc:304] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 300038e5d9694754b6855f62e5e5e679, 8b22fe88c0914cdeb565ee21f8b66cd4; no voters: 
I20251024 20:01:27.682997 21209 raft_consensus.cc:2804] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:01:27.683048 21209 raft_consensus.cc:697] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 1 LEADER]: Becoming Leader. State: Replica: 300038e5d9694754b6855f62e5e5e679, State: Running, Role: LEADER
I20251024 20:01:27.683089 21209 consensus_queue.cc:237] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.683724 20720 catalog_manager.cc:5649] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 reported cstate change: term changed from 0 to 1, leader changed from <none> to 300038e5d9694754b6855f62e5e5e679 (127.18.153.65). New cstate: current_term: 1 leader_uuid: "300038e5d9694754b6855f62e5e5e679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } health_report { overall_health: UNKNOWN } } }
I20251024 20:01:27.718215 21200 consensus_queue.cc:1048] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:01:27.718683 21149 raft_consensus.cc:1275] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 FOLLOWER]: Refusing update from remote peer 300038e5d9694754b6855f62e5e5e679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 20:01:27.718843 21209 consensus_queue.cc:1048] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:01:27.718875 21149 raft_consensus.cc:1275] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 FOLLOWER]: Refusing update from remote peer 300038e5d9694754b6855f62e5e5e679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 20:01:27.719123 21068 raft_consensus.cc:1275] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Refusing update from remote peer 300038e5d9694754b6855f62e5e5e679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 20:01:27.719149 20991 raft_consensus.cc:1275] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 1 FOLLOWER]: Refusing update from remote peer 8b22fe88c0914cdeb565ee21f8b66cd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 20:01:27.719206 21209 consensus_queue.cc:1048] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:01:27.719412 21213 consensus_queue.cc:1048] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:01:27.719543 21068 raft_consensus.cc:1275] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Refusing update from remote peer 300038e5d9694754b6855f62e5e5e679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 20:01:27.719521 21209 consensus_queue.cc:1048] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:01:27.719751 21209 consensus_queue.cc:1048] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:01:27.732492 20991 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2dbef0e1505045468e973459c151f622"
dest_uuid: "300038e5d9694754b6855f62e5e5e679"
 from {username='slave'} at 127.0.0.1:59240
I20251024 20:01:27.732581 20991 raft_consensus.cc:591] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 1 LEADER]: Received request to step down
I20251024 20:01:27.732599 20991 raft_consensus.cc:3055] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 1 LEADER]: Stepping down as leader of term 1
I20251024 20:01:27.732610 20991 raft_consensus.cc:740] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 300038e5d9694754b6855f62e5e5e679, State: Running, Role: LEADER
I20251024 20:01:27.732661 20991 consensus_queue.cc:260] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.732735 20991 raft_consensus.cc:3060] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:01:27.923439 21230 raft_consensus.cc:493] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 300038e5d9694754b6855f62e5e5e679)
I20251024 20:01:27.923548 21230 raft_consensus.cc:515] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.923771 21230 leader_election.cc:290] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:27.926569 20991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "e0594f5433ae41248b6004bf70361182" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "300038e5d9694754b6855f62e5e5e679" is_pre_election: true
I20251024 20:01:27.926677 20991 raft_consensus.cc:2468] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0594f5433ae41248b6004bf70361182 in term 2.
I20251024 20:01:27.926587 21148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "e0594f5433ae41248b6004bf70361182" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" is_pre_election: true
I20251024 20:01:27.926841 21036 leader_election.cc:304] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 300038e5d9694754b6855f62e5e5e679, e0594f5433ae41248b6004bf70361182; no voters: 
I20251024 20:01:27.926827 21148 raft_consensus.cc:2468] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0594f5433ae41248b6004bf70361182 in term 1.
I20251024 20:01:27.926956 21230 raft_consensus.cc:2804] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251024 20:01:27.927006 21230 raft_consensus.cc:493] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 300038e5d9694754b6855f62e5e5e679)
I20251024 20:01:27.927037 21230 raft_consensus.cc:3060] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:01:27.928310 21230 raft_consensus.cc:515] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.928507 21230 leader_election.cc:290] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [CANDIDATE]: Term 2 election: Requested vote from peers 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:27.928551 21148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "e0594f5433ae41248b6004bf70361182" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4"
I20251024 20:01:27.928568 20991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "e0594f5433ae41248b6004bf70361182" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "300038e5d9694754b6855f62e5e5e679"
I20251024 20:01:27.928612 21148 raft_consensus.cc:3060] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:01:27.929181 21148 raft_consensus.cc:2468] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0594f5433ae41248b6004bf70361182 in term 2.
I20251024 20:01:27.929203 20991 raft_consensus.cc:2468] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0594f5433ae41248b6004bf70361182 in term 2.
I20251024 20:01:27.929350 21035 leader_election.cc:304] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b22fe88c0914cdeb565ee21f8b66cd4, e0594f5433ae41248b6004bf70361182; no voters: 
I20251024 20:01:27.929445 21230 raft_consensus.cc:2804] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 2 FOLLOWER]: Leader election won for term 2
I20251024 20:01:27.929491 21230 raft_consensus.cc:697] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 2 LEADER]: Becoming Leader. State: Replica: e0594f5433ae41248b6004bf70361182, State: Running, Role: LEADER
I20251024 20:01:27.929687 21230 consensus_queue.cc:237] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.930120 20716 catalog_manager.cc:5649] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 reported cstate change: term changed from 1 to 2, leader changed from 300038e5d9694754b6855f62e5e5e679 (127.18.153.65) to e0594f5433ae41248b6004bf70361182 (127.18.153.66). New cstate: current_term: 2 leader_uuid: "e0594f5433ae41248b6004bf70361182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } health_report { overall_health: UNKNOWN } } }
I20251024 20:01:27.933686 20991 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2"
dest_uuid: "300038e5d9694754b6855f62e5e5e679"
 from {username='slave'} at 127.0.0.1:59240
I20251024 20:01:27.933765 20991 raft_consensus.cc:591] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 1 LEADER]: Received request to step down
I20251024 20:01:27.933794 20991 raft_consensus.cc:3055] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 1 LEADER]: Stepping down as leader of term 1
I20251024 20:01:27.933812 20991 raft_consensus.cc:740] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 300038e5d9694754b6855f62e5e5e679, State: Running, Role: LEADER
I20251024 20:01:27.933863 20991 consensus_queue.cc:260] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:27.933931 20991 raft_consensus.cc:3060] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:01:28.011601 21148 raft_consensus.cc:1275] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Refusing update from remote peer e0594f5433ae41248b6004bf70361182: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251024 20:01:28.011868 21230 consensus_queue.cc:1048] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:01:28.013123 20991 raft_consensus.cc:1275] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Refusing update from remote peer e0594f5433ae41248b6004bf70361182: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251024 20:01:28.013399 21203 raft_consensus.cc:493] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 300038e5d9694754b6855f62e5e5e679)
I20251024 20:01:28.013458 21203 raft_consensus.cc:515] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.013557 21241 consensus_queue.cc:1048] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:01:28.013571 21203 leader_election.cc:290] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:28.013769 20991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "e0594f5433ae41248b6004bf70361182" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "300038e5d9694754b6855f62e5e5e679" is_pre_election: true
I20251024 20:01:28.013774 21148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "e0594f5433ae41248b6004bf70361182" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" is_pre_election: true
I20251024 20:01:28.013839 20991 raft_consensus.cc:2468] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0594f5433ae41248b6004bf70361182 in term 2.
I20251024 20:01:28.013949 21036 leader_election.cc:304] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 300038e5d9694754b6855f62e5e5e679, e0594f5433ae41248b6004bf70361182; no voters: 
I20251024 20:01:28.014032 21203 raft_consensus.cc:2804] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251024 20:01:28.014067 21203 raft_consensus.cc:493] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 300038e5d9694754b6855f62e5e5e679)
I20251024 20:01:28.014096 21203 raft_consensus.cc:3060] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:01:28.014925 21203 raft_consensus.cc:515] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.015048 21203 leader_election.cc:290] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [CANDIDATE]: Term 2 election: Requested vote from peers 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:28.015264 20991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "e0594f5433ae41248b6004bf70361182" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "300038e5d9694754b6855f62e5e5e679"
I20251024 20:01:28.015290 21148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "e0594f5433ae41248b6004bf70361182" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4"
I20251024 20:01:28.015947 20991 raft_consensus.cc:2468] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0594f5433ae41248b6004bf70361182 in term 2.
I20251024 20:01:28.016109 21036 leader_election.cc:304] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 300038e5d9694754b6855f62e5e5e679, e0594f5433ae41248b6004bf70361182; no voters: 8b22fe88c0914cdeb565ee21f8b66cd4
I20251024 20:01:28.016216 21203 raft_consensus.cc:2804] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 2 FOLLOWER]: Leader election won for term 2
I20251024 20:01:28.016273 21203 raft_consensus.cc:697] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 2 LEADER]: Becoming Leader. State: Replica: e0594f5433ae41248b6004bf70361182, State: Running, Role: LEADER
I20251024 20:01:28.016342 21203 consensus_queue.cc:237] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.016690 20717 catalog_manager.cc:5649] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 reported cstate change: term changed from 1 to 2, leader changed from 300038e5d9694754b6855f62e5e5e679 (127.18.153.65) to e0594f5433ae41248b6004bf70361182 (127.18.153.66). New cstate: current_term: 2 leader_uuid: "e0594f5433ae41248b6004bf70361182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } health_report { overall_health: UNKNOWN } } }
I20251024 20:01:28.089921 21200 raft_consensus.cc:493] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 300038e5d9694754b6855f62e5e5e679)
I20251024 20:01:28.090070 21200 raft_consensus.cc:515] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.090276 21200 leader_election.cc:290] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453), e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339)
I20251024 20:01:28.090484 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182" is_pre_election: true
I20251024 20:01:28.090484 20991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "300038e5d9694754b6855f62e5e5e679" is_pre_election: true
I20251024 20:01:28.090608 20991 raft_consensus.cc:2393] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8b22fe88c0914cdeb565ee21f8b66cd4 in current term 2: Already voted for candidate e0594f5433ae41248b6004bf70361182 in this term.
I20251024 20:01:28.090814 21115 leader_election.cc:304] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b22fe88c0914cdeb565ee21f8b66cd4; no voters: 300038e5d9694754b6855f62e5e5e679, e0594f5433ae41248b6004bf70361182
I20251024 20:01:28.090934 21200 raft_consensus.cc:3060] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:01:28.091722 21200 raft_consensus.cc:2749] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 20:01:28.124862 21148 raft_consensus.cc:1275] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Refusing update from remote peer e0594f5433ae41248b6004bf70361182: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251024 20:01:28.125159 21203 consensus_queue.cc:1048] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:01:28.126688 20991 raft_consensus.cc:1275] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Refusing update from remote peer e0594f5433ae41248b6004bf70361182: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251024 20:01:28.126843 21241 consensus_queue.cc:1048] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:01:28.137697 21070 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2dbef0e1505045468e973459c151f622"
dest_uuid: "e0594f5433ae41248b6004bf70361182"
 from {username='slave'} at 127.0.0.1:49250
I20251024 20:01:28.137777 21070 raft_consensus.cc:591] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 2 LEADER]: Received request to step down
I20251024 20:01:28.137796 21070 raft_consensus.cc:3055] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 2 LEADER]: Stepping down as leader of term 2
I20251024 20:01:28.137809 21070 raft_consensus.cc:740] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e0594f5433ae41248b6004bf70361182, State: Running, Role: LEADER
I20251024 20:01:28.137867 21070 consensus_queue.cc:260] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.137957 21070 raft_consensus.cc:3060] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:01:28.330864 21224 raft_consensus.cc:493] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e0594f5433ae41248b6004bf70361182)
I20251024 20:01:28.330962 21224 raft_consensus.cc:515] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.331207 21224 leader_election.cc:290] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:28.331250 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182" is_pre_election: true
I20251024 20:01:28.331326 21148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" is_pre_election: true
I20251024 20:01:28.331350 21070 raft_consensus.cc:2468] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 300038e5d9694754b6855f62e5e5e679 in term 3.
I20251024 20:01:28.331566 20954 leader_election.cc:304] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 300038e5d9694754b6855f62e5e5e679, e0594f5433ae41248b6004bf70361182; no voters: 
I20251024 20:01:28.331718 21224 raft_consensus.cc:2804] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251024 20:01:28.331759 21224 raft_consensus.cc:493] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e0594f5433ae41248b6004bf70361182)
I20251024 20:01:28.331785 21224 raft_consensus.cc:3060] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:01:28.331982 21200 raft_consensus.cc:493] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e0594f5433ae41248b6004bf70361182)
I20251024 20:01:28.332031 21200 raft_consensus.cc:515] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.332137 21200 leader_election.cc:290] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453), e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339)
I20251024 20:01:28.332295 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182" is_pre_election: true
I20251024 20:01:28.332369 21070 raft_consensus.cc:2468] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b22fe88c0914cdeb565ee21f8b66cd4 in term 3.
I20251024 20:01:28.332394 20991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "300038e5d9694754b6855f62e5e5e679" is_pre_election: true
I20251024 20:01:28.332564 21112 leader_election.cc:304] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b22fe88c0914cdeb565ee21f8b66cd4, e0594f5433ae41248b6004bf70361182; no voters: 
I20251024 20:01:28.332656 21200 raft_consensus.cc:2804] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251024 20:01:28.332708 21200 raft_consensus.cc:493] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e0594f5433ae41248b6004bf70361182)
I20251024 20:01:28.332742 21200 raft_consensus.cc:3060] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:01:28.332785 21224 raft_consensus.cc:515] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.332916 20991 raft_consensus.cc:2393] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8b22fe88c0914cdeb565ee21f8b66cd4 in current term 3: Already voted for candidate 300038e5d9694754b6855f62e5e5e679 in this term.
I20251024 20:01:28.332996 21224 leader_election.cc:290] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 3 election: Requested vote from peers e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:28.333000 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182"
I20251024 20:01:28.333370 21200 raft_consensus.cc:515] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.333472 21200 leader_election.cc:290] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [CANDIDATE]: Term 3 election: Requested vote from peers 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453), e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339)
I20251024 20:01:28.333611 21070 raft_consensus.cc:2468] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 300038e5d9694754b6855f62e5e5e679 in term 3.
I20251024 20:01:28.333664 20991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "300038e5d9694754b6855f62e5e5e679"
I20251024 20:01:28.333715 21148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4"
I20251024 20:01:28.333765 20954 leader_election.cc:304] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 300038e5d9694754b6855f62e5e5e679, e0594f5433ae41248b6004bf70361182; no voters: 
I20251024 20:01:28.333722 20991 raft_consensus.cc:2393] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8b22fe88c0914cdeb565ee21f8b66cd4 in current term 3: Already voted for candidate 300038e5d9694754b6855f62e5e5e679 in this term.
I20251024 20:01:28.333804 21148 raft_consensus.cc:2393] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 300038e5d9694754b6855f62e5e5e679 in current term 3: Already voted for candidate 8b22fe88c0914cdeb565ee21f8b66cd4 in this term.
I20251024 20:01:28.333843 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2dbef0e1505045468e973459c151f622" candidate_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182"
I20251024 20:01:28.333890 21070 raft_consensus.cc:2393] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8b22fe88c0914cdeb565ee21f8b66cd4 in current term 3: Already voted for candidate 300038e5d9694754b6855f62e5e5e679 in this term.
I20251024 20:01:28.333925 21224 raft_consensus.cc:2804] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 3 FOLLOWER]: Leader election won for term 3
I20251024 20:01:28.333972 21224 raft_consensus.cc:697] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 3 LEADER]: Becoming Leader. State: Replica: 300038e5d9694754b6855f62e5e5e679, State: Running, Role: LEADER
I20251024 20:01:28.333998 21112 leader_election.cc:304] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b22fe88c0914cdeb565ee21f8b66cd4; no voters: 300038e5d9694754b6855f62e5e5e679, e0594f5433ae41248b6004bf70361182
I20251024 20:01:28.334064 21200 raft_consensus.cc:2749] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251024 20:01:28.334025 21224 consensus_queue.cc:237] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.334520 20716 catalog_manager.cc:5649] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 reported cstate change: term changed from 2 to 3, leader changed from e0594f5433ae41248b6004bf70361182 (127.18.153.66) to 300038e5d9694754b6855f62e5e5e679 (127.18.153.65). New cstate: current_term: 3 leader_uuid: "300038e5d9694754b6855f62e5e5e679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } health_report { overall_health: UNKNOWN } } }
I20251024 20:01:28.338599 21070 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2"
dest_uuid: "e0594f5433ae41248b6004bf70361182"
 from {username='slave'} at 127.0.0.1:49250
I20251024 20:01:28.338668 21070 raft_consensus.cc:591] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 2 LEADER]: Received request to step down
I20251024 20:01:28.338698 21070 raft_consensus.cc:3055] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 2 LEADER]: Stepping down as leader of term 2
I20251024 20:01:28.338724 21070 raft_consensus.cc:740] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e0594f5433ae41248b6004bf70361182, State: Running, Role: LEADER
I20251024 20:01:28.338774 21070 consensus_queue.cc:260] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.338847 21070 raft_consensus.cc:3060] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:01:28.429800 21148 raft_consensus.cc:1275] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 3 FOLLOWER]: Refusing update from remote peer 300038e5d9694754b6855f62e5e5e679: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251024 20:01:28.430083 21224 consensus_queue.cc:1048] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251024 20:01:28.431025 21070 raft_consensus.cc:1275] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Refusing update from remote peer 300038e5d9694754b6855f62e5e5e679: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251024 20:01:28.431268 21242 consensus_queue.cc:1048] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251024 20:01:28.444720 20904 catalog_manager.cc:1269] Loaded cluster ID: bbb04eb8879d4aa8a0e7ff573f23a582
I20251024 20:01:28.444798 20904 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a: loading cluster ID for follower catalog manager: success
I20251024 20:01:28.445489 20904 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a: acquiring CA information for follower catalog manager: success
I20251024 20:01:28.445605 20904 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251024 20:01:28.484676 21242 raft_consensus.cc:493] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e0594f5433ae41248b6004bf70361182)
I20251024 20:01:28.484809 21242 raft_consensus.cc:515] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.484998 21242 leader_election.cc:290] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:28.485191 21148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" is_pre_election: true
I20251024 20:01:28.485191 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182" is_pre_election: true
I20251024 20:01:28.485338 21070 raft_consensus.cc:2468] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 300038e5d9694754b6855f62e5e5e679 in term 3.
I20251024 20:01:28.485497 20954 leader_election.cc:304] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 300038e5d9694754b6855f62e5e5e679, e0594f5433ae41248b6004bf70361182; no voters: 8b22fe88c0914cdeb565ee21f8b66cd4
I20251024 20:01:28.485615 21242 raft_consensus.cc:2804] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251024 20:01:28.485662 21242 raft_consensus.cc:493] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e0594f5433ae41248b6004bf70361182)
I20251024 20:01:28.485692 21242 raft_consensus.cc:3060] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:01:28.486408 21242 raft_consensus.cc:515] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.486534 21242 leader_election.cc:290] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 3 election: Requested vote from peers e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:28.486703 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182"
I20251024 20:01:28.486682 21148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4"
I20251024 20:01:28.487435 21070 raft_consensus.cc:2468] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 300038e5d9694754b6855f62e5e5e679 in term 3.
I20251024 20:01:28.487591 20954 leader_election.cc:304] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 300038e5d9694754b6855f62e5e5e679, e0594f5433ae41248b6004bf70361182; no voters: 8b22fe88c0914cdeb565ee21f8b66cd4
I20251024 20:01:28.487676 21242 raft_consensus.cc:2804] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 3 FOLLOWER]: Leader election won for term 3
I20251024 20:01:28.487739 21242 raft_consensus.cc:697] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 3 LEADER]: Becoming Leader. State: Replica: 300038e5d9694754b6855f62e5e5e679, State: Running, Role: LEADER
I20251024 20:01:28.487799 21242 consensus_queue.cc:237] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.488204 20716 catalog_manager.cc:5649] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 reported cstate change: term changed from 2 to 3, leader changed from e0594f5433ae41248b6004bf70361182 (127.18.153.66) to 300038e5d9694754b6855f62e5e5e679 (127.18.153.65). New cstate: current_term: 3 leader_uuid: "300038e5d9694754b6855f62e5e5e679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } health_report { overall_health: UNKNOWN } } }
I20251024 20:01:28.525789 21200 raft_consensus.cc:493] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e0594f5433ae41248b6004bf70361182)
I20251024 20:01:28.525880 21200 raft_consensus.cc:515] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.526005 21200 leader_election.cc:290] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453), e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339)
I20251024 20:01:28.526206 20991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "300038e5d9694754b6855f62e5e5e679" is_pre_election: true
I20251024 20:01:28.526227 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "784c0490ffca4ccaa5cece7bae6064c2" candidate_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182" is_pre_election: true
I20251024 20:01:28.526310 21070 raft_consensus.cc:2393] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8b22fe88c0914cdeb565ee21f8b66cd4 in current term 3: Already voted for candidate 300038e5d9694754b6855f62e5e5e679 in this term.
I20251024 20:01:28.526455 21112 leader_election.cc:304] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b22fe88c0914cdeb565ee21f8b66cd4; no voters: 300038e5d9694754b6855f62e5e5e679, e0594f5433ae41248b6004bf70361182
I20251024 20:01:28.526557 21200 raft_consensus.cc:3060] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:01:28.527264 21200 raft_consensus.cc:2749] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251024 20:01:28.542572 21148 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a4e6dc18db1e49208af2fa5dfaae96a6"
dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4"
 from {username='slave'} at 127.0.0.1:37258
I20251024 20:01:28.542650 21148 raft_consensus.cc:591] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 LEADER]: Received request to step down
I20251024 20:01:28.542668 21148 raft_consensus.cc:3055] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 LEADER]: Stepping down as leader of term 1
I20251024 20:01:28.542681 21148 raft_consensus.cc:740] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b22fe88c0914cdeb565ee21f8b66cd4, State: Running, Role: LEADER
I20251024 20:01:28.542730 21148 consensus_queue.cc:260] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.542800 21148 raft_consensus.cc:3060] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:01:28.569993 21148 raft_consensus.cc:1275] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 3 FOLLOWER]: Refusing update from remote peer 300038e5d9694754b6855f62e5e5e679: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251024 20:01:28.570264 21209 consensus_queue.cc:1048] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251024 20:01:28.571269 21070 raft_consensus.cc:1275] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Refusing update from remote peer 300038e5d9694754b6855f62e5e5e679: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251024 20:01:28.571506 21242 consensus_queue.cc:1048] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251024 20:01:28.702852 21224 raft_consensus.cc:493] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b22fe88c0914cdeb565ee21f8b66cd4)
I20251024 20:01:28.702982 21224 raft_consensus.cc:515] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.703193 21224 leader_election.cc:290] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:28.703449 21148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e6dc18db1e49208af2fa5dfaae96a6" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" is_pre_election: true
I20251024 20:01:28.703473 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e6dc18db1e49208af2fa5dfaae96a6" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182" is_pre_election: true
I20251024 20:01:28.703521 21148 raft_consensus.cc:2468] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 300038e5d9694754b6855f62e5e5e679 in term 2.
I20251024 20:01:28.703727 20956 leader_election.cc:304] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 300038e5d9694754b6855f62e5e5e679, 8b22fe88c0914cdeb565ee21f8b66cd4; no voters: 
I20251024 20:01:28.703809 21224 raft_consensus.cc:2804] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251024 20:01:28.703850 21224 raft_consensus.cc:493] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b22fe88c0914cdeb565ee21f8b66cd4)
I20251024 20:01:28.703869 21224 raft_consensus.cc:3060] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:01:28.704510 21224 raft_consensus.cc:515] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.704620 21224 leader_election.cc:290] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 2 election: Requested vote from peers e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:28.704768 21070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e6dc18db1e49208af2fa5dfaae96a6" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e0594f5433ae41248b6004bf70361182"
I20251024 20:01:28.704818 21148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e6dc18db1e49208af2fa5dfaae96a6" candidate_uuid: "300038e5d9694754b6855f62e5e5e679" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4"
I20251024 20:01:28.705437 21148 raft_consensus.cc:2468] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 300038e5d9694754b6855f62e5e5e679 in term 2.
I20251024 20:01:28.705595 20956 leader_election.cc:304] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 300038e5d9694754b6855f62e5e5e679, 8b22fe88c0914cdeb565ee21f8b66cd4; no voters: e0594f5433ae41248b6004bf70361182
I20251024 20:01:28.705677 21224 raft_consensus.cc:2804] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Leader election won for term 2
I20251024 20:01:28.705729 21224 raft_consensus.cc:697] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 2 LEADER]: Becoming Leader. State: Replica: 300038e5d9694754b6855f62e5e5e679, State: Running, Role: LEADER
I20251024 20:01:28.705782 21224 consensus_queue.cc:237] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.706205 20716 catalog_manager.cc:5649] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 reported cstate change: term changed from 1 to 2, leader changed from 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67) to 300038e5d9694754b6855f62e5e5e679 (127.18.153.65). New cstate: current_term: 2 leader_uuid: "300038e5d9694754b6855f62e5e5e679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } health_report { overall_health: UNKNOWN } } }
I20251024 20:01:28.743016 21230 raft_consensus.cc:493] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b22fe88c0914cdeb565ee21f8b66cd4)
I20251024 20:01:28.744715 20716 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49300:
table { table_id: "ef884b5c6c8b4ed6bc23772e0adeb300" table_name: "test-table" } new_table_name: "NewTableName"
I20251024 20:01:28.744757 21230 raft_consensus.cc:515] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 } } peers { permanent_uuid: "300038e5d9694754b6855f62e5e5e679" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 33453 } } peers { permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 } }
I20251024 20:01:28.745172 21148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e6dc18db1e49208af2fa5dfaae96a6" candidate_uuid: "e0594f5433ae41248b6004bf70361182" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" is_pre_election: true
I20251024 20:01:28.745247 21148 raft_consensus.cc:2393] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e0594f5433ae41248b6004bf70361182 in current term 2: Already voted for candidate 300038e5d9694754b6855f62e5e5e679 in this term.
I20251024 20:01:28.745286 20991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e6dc18db1e49208af2fa5dfaae96a6" candidate_uuid: "e0594f5433ae41248b6004bf70361182" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "300038e5d9694754b6855f62e5e5e679" is_pre_election: true
I20251024 20:01:28.745595 21035 leader_election.cc:304] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0594f5433ae41248b6004bf70361182; no voters: 300038e5d9694754b6855f62e5e5e679, 8b22fe88c0914cdeb565ee21f8b66cd4
I20251024 20:01:28.745687 21203 raft_consensus.cc:3060] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:01:28.746639 21203 raft_consensus.cc:2749] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251024 20:01:28.746719 20716 catalog_manager.cc:5101] could not convert tablet's a4e6dc18db1e49208af2fa5dfaae96a6 schema from PB: OK
I20251024 20:01:28.747133 21230 leader_election.cc:290] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453), 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:28.747579 21148 raft_consensus.cc:1275] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Refusing update from remote peer 300038e5d9694754b6855f62e5e5e679: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251024 20:01:28.747887 21070 raft_consensus.cc:1275] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 2 FOLLOWER]: Refusing update from remote peer 300038e5d9694754b6855f62e5e5e679: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251024 20:01:28.748127 19045 master.cc:561] Master@127.18.153.126:38517 shutting down...
I20251024 20:01:28.748237 21209 consensus_queue.cc:1048] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0594f5433ae41248b6004bf70361182" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 38339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:01:28.748405 21246 consensus_queue.cc:1048] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:01:28.749555 21251 tablet.cc:1722] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679: Alter schema from (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20251024 20:01:28.749881 21250 tablet.cc:1722] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679: Alter schema from (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20251024 20:01:28.750151 21252 tablet.cc:1722] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4: Alter schema from (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20251024 20:01:28.751394 21253 tablet.cc:1722] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4: Alter schema from (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20251024 20:01:28.751822 21250 tablet.cc:1722] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679: Alter schema from (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20251024 20:01:28.751981 21255 tablet.cc:1722] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182: Alter schema from (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20251024 20:01:28.752101 21257 tablet.cc:1722] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182: Alter schema from (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20251024 20:01:28.754070 21255 tablet.cc:1722] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182: Alter schema from (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20251024 20:01:28.755388 21252 tablet.cc:1722] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4: Alter schema from (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    0:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20251024 20:01:28.758647 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:01:28.758730 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:28.758752 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a: stopping tablet replica
I20251024 20:01:28.771118 19045 master.cc:583] Master@127.18.153.126:38517 shutdown complete.
I20251024 20:01:28.947222 21262 raft_consensus.cc:493] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a339d74b39234a11a469995395989e7a)
I20251024 20:01:28.947347 21262 raft_consensus.cc:515] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:28.947563 21262 leader_election.cc:290] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a339d74b39234a11a469995395989e7a (127.18.153.126:38517), 496a3c55c0994110b7824e8e0ea4015a (127.18.153.124:38423)
I20251024 20:01:28.947707 21263 raft_consensus.cc:493] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a339d74b39234a11a469995395989e7a)
I20251024 20:01:28.947763 21263 raft_consensus.cc:515] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:28.947793 20862 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df8fdb366ca94c638b9886b02d552ed7" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "496a3c55c0994110b7824e8e0ea4015a" is_pre_election: true
I20251024 20:01:28.947845 20862 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8fdb366ca94c638b9886b02d552ed7 in term 1.
I20251024 20:01:28.948156 20771 leader_election.cc:304] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 496a3c55c0994110b7824e8e0ea4015a, df8fdb366ca94c638b9886b02d552ed7; no voters: 
I20251024 20:01:28.948173 21263 leader_election.cc:290] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a339d74b39234a11a469995395989e7a (127.18.153.126:38517), df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773)
I20251024 20:01:28.948320 21262 raft_consensus.cc:2804] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251024 20:01:28.948377 21262 raft_consensus.cc:493] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a339d74b39234a11a469995395989e7a)
I20251024 20:01:28.948333 20796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "496a3c55c0994110b7824e8e0ea4015a" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "df8fdb366ca94c638b9886b02d552ed7" is_pre_election: true
I20251024 20:01:28.948398 21262 raft_consensus.cc:3060] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 1 FOLLOWER]: Advancing to term 2
W20251024 20:01:28.948532 20772 leader_election.cc:336] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a339d74b39234a11a469995395989e7a (127.18.153.126:38517): Network error: Client connection negotiation failed: client connection to 127.18.153.126:38517: connect: Connection refused (error 111)
I20251024 20:01:28.949141 21262 raft_consensus.cc:515] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:28.949256 20796 raft_consensus.cc:2393] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 496a3c55c0994110b7824e8e0ea4015a in current term 2: Already voted for candidate df8fdb366ca94c638b9886b02d552ed7 in this term.
I20251024 20:01:28.949271 21262 leader_election.cc:290] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [CANDIDATE]: Term 2 election: Requested vote from peers a339d74b39234a11a469995395989e7a (127.18.153.126:38517), 496a3c55c0994110b7824e8e0ea4015a (127.18.153.124:38423)
W20251024 20:01:28.949421 20838 leader_election.cc:336] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a339d74b39234a11a469995395989e7a (127.18.153.126:38517): Network error: Client connection negotiation failed: client connection to 127.18.153.126:38517: connect: Connection refused (error 111)
I20251024 20:01:28.949474 20836 leader_election.cc:304] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 496a3c55c0994110b7824e8e0ea4015a; no voters: a339d74b39234a11a469995395989e7a, df8fdb366ca94c638b9886b02d552ed7
I20251024 20:01:28.949529 20862 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df8fdb366ca94c638b9886b02d552ed7" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "496a3c55c0994110b7824e8e0ea4015a"
I20251024 20:01:28.949558 21263 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 1 FOLLOWER]: Advancing to term 2
W20251024 20:01:28.949787 20772 leader_election.cc:336] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a339d74b39234a11a469995395989e7a (127.18.153.126:38517): Network error: Client connection negotiation failed: client connection to 127.18.153.126:38517: connect: Connection refused (error 111)
I20251024 20:01:28.950222 21263 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 20:01:28.950784 20862 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8fdb366ca94c638b9886b02d552ed7 in term 2.
I20251024 20:01:28.950945 20771 leader_election.cc:304] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 496a3c55c0994110b7824e8e0ea4015a, df8fdb366ca94c638b9886b02d552ed7; no voters: a339d74b39234a11a469995395989e7a
I20251024 20:01:28.951030 21262 raft_consensus.cc:2804] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 2 FOLLOWER]: Leader election won for term 2
I20251024 20:01:28.951159 21262 raft_consensus.cc:697] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 2 LEADER]: Becoming Leader. State: Replica: df8fdb366ca94c638b9886b02d552ed7, State: Running, Role: LEADER
I20251024 20:01:28.951203 21262 consensus_queue.cc:237] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:28.951428 21268 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df8fdb366ca94c638b9886b02d552ed7. Latest consensus state: current_term: 2 leader_uuid: "df8fdb366ca94c638b9886b02d552ed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:28.951503 21268 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:28.951639 21269 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:01:28.951857 21269 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300]
I20251024 20:01:28.951994 21269 tablet_loader.cc:96] loaded metadata for tablet 2dbef0e1505045468e973459c151f622 (table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300])
I20251024 20:01:28.952039 21269 tablet_loader.cc:96] loaded metadata for tablet 784c0490ffca4ccaa5cece7bae6064c2 (table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300])
I20251024 20:01:28.952080 21269 tablet_loader.cc:96] loaded metadata for tablet a4e6dc18db1e49208af2fa5dfaae96a6 (table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300])
I20251024 20:01:28.952116 21269 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:01:28.952183 21269 catalog_manager.cc:1269] Loaded cluster ID: bbb04eb8879d4aa8a0e7ff573f23a582
I20251024 20:01:28.952208 21269 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:01:28.952394 21269 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:01:28.952458 21269 catalog_manager.cc:6033] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: Loaded TSK: 0
I20251024 20:01:28.952569 21269 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:01:28.972535 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:28.973906 21273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:28.973994 21271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:28.974035 21270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:28.974251 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:28.974372 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:28.974403 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:28.974421 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336088974420 us; error 0 us; skew 500 ppm
I20251024 20:01:28.975137 19045 webserver.cc:492] Webserver started at http://127.18.153.126:35241/ using document root <none> and password file <none>
I20251024 20:01:28.975219 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:28.975256 19045 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251024 20:01:28.975879 19045 fs_manager.cc:714] Time spent opening directory manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:28.976378 21278 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:28.976503 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:28.976545 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-0-root
uuid: "a339d74b39234a11a469995395989e7a"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:28.976601 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:28.982244 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:28.982429 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:28.985200 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.126:38517
I20251024 20:01:28.985888 21330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.126:38517 every 8 connection(s)
I20251024 20:01:28.986478 21331 sys_catalog.cc:263] Verifying existing consensus state
I20251024 20:01:28.986788 21331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a: Bootstrap starting.
I20251024 20:01:28.988711 21331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=7 ignored=0} mutations{seen=12 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251024 20:01:28.988916 21331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a: Bootstrap complete.
I20251024 20:01:28.989097 21331 raft_consensus.cc:359] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:28.989161 21331 raft_consensus.cc:740] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a339d74b39234a11a469995395989e7a, State: Initialized, Role: FOLLOWER
I20251024 20:01:28.989199 21331 consensus_queue.cc:260] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:28.989372 21331 sys_catalog.cc:565] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:01:28.989372 21334 sys_catalog.cc:455] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:28.989457 21334 sys_catalog.cc:458] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:28.990610 21345 catalog_manager.cc:1269] Loaded cluster ID: bbb04eb8879d4aa8a0e7ff573f23a582
I20251024 20:01:28.990661 21345 catalog_manager.cc:1562] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a: loading cluster ID for follower catalog manager: success
I20251024 20:01:28.991425 21345 catalog_manager.cc:1584] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a: acquiring CA information for follower catalog manager: success
I20251024 20:01:28.991568 21345 catalog_manager.cc:1612] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251024 20:01:29.036134 21305 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:01:29.036896 21305 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 2 FOLLOWER]: Refusing update from remote peer df8fdb366ca94c638b9886b02d552ed7: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20251024 20:01:29.037161 21268 consensus_queue.cc:1048] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20251024 20:01:29.037750 21334 sys_catalog.cc:455] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader df8fdb366ca94c638b9886b02d552ed7. Latest consensus state: current_term: 2 leader_uuid: "df8fdb366ca94c638b9886b02d552ed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.037822 21334 sys_catalog.cc:458] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:29.038148 21262 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "df8fdb366ca94c638b9886b02d552ed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.038233 21262 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:29.038313 20862 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 2 FOLLOWER]: Refusing update from remote peer df8fdb366ca94c638b9886b02d552ed7: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20251024 20:01:29.038409 21334 sys_catalog.cc:455] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "df8fdb366ca94c638b9886b02d552ed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.038473 21334 sys_catalog.cc:458] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:29.038630 21262 consensus_queue.cc:1048] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20251024 20:01:29.039037 21263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: SysCatalogTable state changed. Reason: New leader df8fdb366ca94c638b9886b02d552ed7. Latest consensus state: current_term: 2 leader_uuid: "df8fdb366ca94c638b9886b02d552ed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.039067 21350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "df8fdb366ca94c638b9886b02d552ed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.039129 21263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:29.039146 21350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:29.056612 21019 heartbeater.cc:499] Master 127.18.153.125:38773 was elected leader, sending a full tablet report...
I20251024 20:01:29.056850 21098 heartbeater.cc:499] Master 127.18.153.125:38773 was elected leader, sending a full tablet report...
I20251024 20:01:29.057804 21262 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "df8fdb366ca94c638b9886b02d552ed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.057915 21262 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:29.058017 21177 heartbeater.cc:499] Master 127.18.153.125:38773 was elected leader, sending a full tablet report...
I20251024 20:01:29.059463 20786 catalog_manager.cc:6194] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300] alter complete (version 1)
I20251024 20:01:29.060091 21018 heartbeater.cc:344] Connected to a master server at 127.18.153.126:38517
I20251024 20:01:29.060750 21295 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "300038e5d9694754b6855f62e5e5e679" instance_seqno: 1761336087510224) as {username='slave'} at 127.0.0.1:49432; Asking this server to re-register.
I20251024 20:01:29.060879 21097 heartbeater.cc:344] Connected to a master server at 127.18.153.126:38517
I20251024 20:01:29.061028 21018 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:29.061198 21295 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e0594f5433ae41248b6004bf70361182" instance_seqno: 1761336087554234) as {username='slave'} at 127.0.0.1:49436; Asking this server to re-register.
I20251024 20:01:29.061302 21294 ts_manager.cc:194] Registered new tserver with Master: 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453)
I20251024 20:01:29.061473 21097 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:29.061585 21176 heartbeater.cc:344] Connected to a master server at 127.18.153.126:38517
I20251024 20:01:29.061765 21294 ts_manager.cc:194] Registered new tserver with Master: e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339)
I20251024 20:01:29.061866 21295 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" instance_seqno: 1761336087573244) as {username='slave'} at 127.0.0.1:49446; Asking this server to re-register.
I20251024 20:01:29.062077 21176 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:29.062288 21295 ts_manager.cc:194] Registered new tserver with Master: 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:29.062801 19045 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.077143072s
I20251024 20:01:29.063069 19045 master.cc:561] Master@127.18.153.125:38773 shutting down...
I20251024 20:01:29.065675 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 2 LEADER]: Raft consensus shutting down.
I20251024 20:01:29.065759 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:29.065843 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: stopping tablet replica
I20251024 20:01:29.078299 19045 master.cc:583] Master@127.18.153.125:38773 shutdown complete.
I20251024 20:01:29.260004 21334 raft_consensus.cc:493] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df8fdb366ca94c638b9886b02d552ed7)
I20251024 20:01:29.260136 21334 raft_consensus.cc:515] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.260376 21334 leader_election.cc:290] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773), 496a3c55c0994110b7824e8e0ea4015a (127.18.153.124:38423)
I20251024 20:01:29.260591 21350 raft_consensus.cc:493] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df8fdb366ca94c638b9886b02d552ed7)
I20251024 20:01:29.260654 21350 raft_consensus.cc:515] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
W20251024 20:01:29.261194 21280 leader_election.cc:336] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773): Network error: Client connection negotiation failed: client connection to 127.18.153.125:38773: connect: Connection refused (error 111)
I20251024 20:01:29.261260 21350 leader_election.cc:290] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a339d74b39234a11a469995395989e7a (127.18.153.126:38517), df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773)
W20251024 20:01:29.261349 20836 leader_election.cc:336] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773): Network error: Client connection negotiation failed: client connection to 127.18.153.125:38773: connect: Connection refused (error 111)
I20251024 20:01:29.264066 21305 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "496a3c55c0994110b7824e8e0ea4015a" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "a339d74b39234a11a469995395989e7a" is_pre_election: true
I20251024 20:01:29.264191 21305 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 496a3c55c0994110b7824e8e0ea4015a in term 2.
I20251024 20:01:29.264066 20862 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a339d74b39234a11a469995395989e7a" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "496a3c55c0994110b7824e8e0ea4015a" is_pre_election: true
I20251024 20:01:29.264398 20862 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a339d74b39234a11a469995395989e7a in term 2.
I20251024 20:01:29.264328 20838 leader_election.cc:304] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 496a3c55c0994110b7824e8e0ea4015a, a339d74b39234a11a469995395989e7a; no voters: df8fdb366ca94c638b9886b02d552ed7
I20251024 20:01:29.264619 21281 leader_election.cc:304] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 496a3c55c0994110b7824e8e0ea4015a, a339d74b39234a11a469995395989e7a; no voters: df8fdb366ca94c638b9886b02d552ed7
I20251024 20:01:29.264788 21334 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251024 20:01:29.264837 21334 raft_consensus.cc:493] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 2 FOLLOWER]: Starting leader election (detected failure of leader df8fdb366ca94c638b9886b02d552ed7)
I20251024 20:01:29.264864 21334 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 2 FOLLOWER]: Advancing to term 3
I20251024 20:01:29.265681 21334 raft_consensus.cc:515] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.265825 21334 leader_election.cc:290] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [CANDIDATE]: Term 3 election: Requested vote from peers df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773), 496a3c55c0994110b7824e8e0ea4015a (127.18.153.124:38423)
I20251024 20:01:29.265955 20862 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a339d74b39234a11a469995395989e7a" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "496a3c55c0994110b7824e8e0ea4015a"
I20251024 20:01:29.264638 21350 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251024 20:01:29.266253 20862 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 2 FOLLOWER]: Advancing to term 3
W20251024 20:01:29.266727 21280 leader_election.cc:336] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773): Network error: Client connection negotiation failed: client connection to 127.18.153.125:38773: connect: Connection refused (error 111)
I20251024 20:01:29.267027 20862 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a339d74b39234a11a469995395989e7a in term 3.
I20251024 20:01:29.267099 21350 raft_consensus.cc:493] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 20:01:29.267158 21350 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 3 FOLLOWER]: Advancing to term 4
I20251024 20:01:29.267197 21281 leader_election.cc:304] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 496a3c55c0994110b7824e8e0ea4015a, a339d74b39234a11a469995395989e7a; no voters: df8fdb366ca94c638b9886b02d552ed7
I20251024 20:01:29.267282 21334 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 3 FOLLOWER]: Leader election won for term 3
I20251024 20:01:29.267482 21334 raft_consensus.cc:697] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 3 LEADER]: Becoming Leader. State: Replica: a339d74b39234a11a469995395989e7a, State: Running, Role: LEADER
I20251024 20:01:29.267539 21334 consensus_queue.cc:237] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 2.17, Last appended by leader: 17, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.267748 21359 sys_catalog.cc:455] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a339d74b39234a11a469995395989e7a. Latest consensus state: current_term: 3 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.267830 21359 sys_catalog.cc:458] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:29.267920 21350 raft_consensus.cc:515] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.268011 21360 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:01:29.268167 21350 leader_election.cc:290] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [CANDIDATE]: Term 4 election: Requested vote from peers a339d74b39234a11a469995395989e7a (127.18.153.126:38517), df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773)
I20251024 20:01:29.268245 21360 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300]
I20251024 20:01:29.268345 21360 tablet_loader.cc:96] loaded metadata for tablet 2dbef0e1505045468e973459c151f622 (table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300])
I20251024 20:01:29.268395 21360 tablet_loader.cc:96] loaded metadata for tablet 784c0490ffca4ccaa5cece7bae6064c2 (table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300])
I20251024 20:01:29.268435 21360 tablet_loader.cc:96] loaded metadata for tablet a4e6dc18db1e49208af2fa5dfaae96a6 (table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300])
I20251024 20:01:29.268474 21360 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:01:29.268532 21305 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "496a3c55c0994110b7824e8e0ea4015a" candidate_term: 4 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "a339d74b39234a11a469995395989e7a"
W20251024 20:01:29.268539 20836 leader_election.cc:336] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773): Network error: Client connection negotiation failed: client connection to 127.18.153.125:38773: connect: Connection refused (error 111)
I20251024 20:01:29.268548 21360 catalog_manager.cc:1269] Loaded cluster ID: bbb04eb8879d4aa8a0e7ff573f23a582
I20251024 20:01:29.268623 21360 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:01:29.268723 20838 leader_election.cc:304] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 496a3c55c0994110b7824e8e0ea4015a; no voters: a339d74b39234a11a469995395989e7a, df8fdb366ca94c638b9886b02d552ed7
I20251024 20:01:29.268805 21360 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:01:29.268812 21350 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251024 20:01:29.268908 21360 catalog_manager.cc:6033] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a: Loaded TSK: 0
I20251024 20:01:29.268983 21360 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:01:29.280081 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:29.281466 21362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:29.281490 21361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:29.281553 21364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:29.281836 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:29.281957 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:29.281985 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:29.282008 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336089282008 us; error 0 us; skew 500 ppm
I20251024 20:01:29.282740 19045 webserver.cc:492] Webserver started at http://127.18.153.125:40191/ using document root <none> and password file <none>
I20251024 20:01:29.282830 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:29.282869 19045 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251024 20:01:29.283511 19045 fs_manager.cc:714] Time spent opening directory manager: real 0.000s	user 0.001s	sys 0.000s
I20251024 20:01:29.283983 21369 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:29.284119 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:29.284168 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-1-root
uuid: "df8fdb366ca94c638b9886b02d552ed7"
format_stamp: "Formatted at 2025-10-24 20:01:27 on dist-test-slave-375r"
I20251024 20:01:29.284232 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:29.293367 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:29.293573 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:29.296216 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.125:38773
I20251024 20:01:29.296254 21421 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.125:38773 every 8 connection(s)
I20251024 20:01:29.297006 21422 sys_catalog.cc:263] Verifying existing consensus state
I20251024 20:01:29.297329 21422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: Bootstrap starting.
I20251024 20:01:29.299237 21422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=0} inserts{seen=7 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251024 20:01:29.299494 21422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: Bootstrap complete.
I20251024 20:01:29.299692 21422 raft_consensus.cc:359] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.299757 21422 raft_consensus.cc:740] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8fdb366ca94c638b9886b02d552ed7, State: Initialized, Role: FOLLOWER
I20251024 20:01:29.299806 21422 consensus_queue.cc:260] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 2.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.299978 21422 sys_catalog.cc:565] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:01:29.299991 21425 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.300109 21425 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:29.301172 21436 catalog_manager.cc:1269] Loaded cluster ID: bbb04eb8879d4aa8a0e7ff573f23a582
I20251024 20:01:29.301218 21436 catalog_manager.cc:1562] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: loading cluster ID for follower catalog manager: success
I20251024 20:01:29.301882 21436 catalog_manager.cc:1584] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: acquiring CA information for follower catalog manager: success
I20251024 20:01:29.302016 21436 catalog_manager.cc:1612] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251024 20:01:29.350471 20862 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 4 FOLLOWER]: Rejecting Update request from peer a339d74b39234a11a469995395989e7a for earlier term 3. Current term is 4. Ops: []
I20251024 20:01:29.350740 21359 consensus_queue.cc:1059] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20251024 20:01:29.350871 21359 raft_consensus.cc:3055] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 3 LEADER]: Stepping down as leader of term 3
I20251024 20:01:29.350896 21359 raft_consensus.cc:740] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a339d74b39234a11a469995395989e7a, State: Running, Role: LEADER
I20251024 20:01:29.350966 21359 consensus_queue.cc:260] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 3.18, Last appended by leader: 18, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.351047 21359 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 3 FOLLOWER]: Advancing to term 4
I20251024 20:01:29.368948 21019 heartbeater.cc:344] Connected to a master server at 127.18.153.125:38773
I20251024 20:01:29.368997 21177 heartbeater.cc:344] Connected to a master server at 127.18.153.125:38773
I20251024 20:01:29.369195 21098 heartbeater.cc:344] Connected to a master server at 127.18.153.125:38773
I20251024 20:01:29.369271 21386 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "8b22fe88c0914cdeb565ee21f8b66cd4" instance_seqno: 1761336087573244) as {username='slave'} at 127.0.0.1:39432; Asking this server to re-register.
I20251024 20:01:29.369371 21385 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "300038e5d9694754b6855f62e5e5e679" instance_seqno: 1761336087510224) as {username='slave'} at 127.0.0.1:39452; Asking this server to re-register.
I20251024 20:01:29.369488 21386 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e0594f5433ae41248b6004bf70361182" instance_seqno: 1761336087554234) as {username='slave'} at 127.0.0.1:39438; Asking this server to re-register.
I20251024 20:01:29.369525 21019 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:29.369536 21177 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:29.369622 21098 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:29.369782 21386 ts_manager.cc:194] Registered new tserver with Master: 300038e5d9694754b6855f62e5e5e679 (127.18.153.65:33453)
I20251024 20:01:29.369844 21385 ts_manager.cc:194] Registered new tserver with Master: e0594f5433ae41248b6004bf70361182 (127.18.153.66:38339)
I20251024 20:01:29.369859 21384 ts_manager.cc:194] Registered new tserver with Master: 8b22fe88c0914cdeb565ee21f8b66cd4 (127.18.153.67:45221)
I20251024 20:01:29.370026 19045 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.073560558s
I20251024 20:01:29.494062 21425 raft_consensus.cc:493] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 20:01:29.494190 21425 raft_consensus.cc:515] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.494405 21425 leader_election.cc:290] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a339d74b39234a11a469995395989e7a (127.18.153.126:38517), 496a3c55c0994110b7824e8e0ea4015a (127.18.153.124:38423)
I20251024 20:01:29.497993 20862 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df8fdb366ca94c638b9886b02d552ed7" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "496a3c55c0994110b7824e8e0ea4015a" is_pre_election: true
I20251024 20:01:29.497993 21305 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df8fdb366ca94c638b9886b02d552ed7" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "a339d74b39234a11a469995395989e7a" is_pre_election: true
I20251024 20:01:29.498184 20862 raft_consensus.cc:2368] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df8fdb366ca94c638b9886b02d552ed7 for earlier term 3. Current term is 4.
I20251024 20:01:29.498225 21305 raft_consensus.cc:2368] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df8fdb366ca94c638b9886b02d552ed7 for earlier term 3. Current term is 4.
I20251024 20:01:29.498343 21372 leader_election.cc:400] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 496a3c55c0994110b7824e8e0ea4015a (127.18.153.124:38423) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df8fdb366ca94c638b9886b02d552ed7 for earlier term 3. Current term is 4.
I20251024 20:01:29.498392 21372 leader_election.cc:403] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251024 20:01:29.498466 21425 raft_consensus.cc:3060] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 2 FOLLOWER]: Advancing to term 4
I20251024 20:01:29.498541 21373 leader_election.cc:400] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a339d74b39234a11a469995395989e7a (127.18.153.126:38517) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df8fdb366ca94c638b9886b02d552ed7 for earlier term 3. Current term is 4.
I20251024 20:01:29.499190 21425 raft_consensus.cc:2749] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 496a3c55c0994110b7824e8e0ea4015a (127.18.153.124:38423) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df8fdb366ca94c638b9886b02d552ed7 for earlier term 3. Current term is 4.
I20251024 20:01:29.524667 21350 raft_consensus.cc:493] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 20:01:29.524778 21350 raft_consensus.cc:515] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.524964 21350 leader_election.cc:290] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a339d74b39234a11a469995395989e7a (127.18.153.126:38517), df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773)
I20251024 20:01:29.525179 21305 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "496a3c55c0994110b7824e8e0ea4015a" candidate_term: 5 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "a339d74b39234a11a469995395989e7a" is_pre_election: true
I20251024 20:01:29.525311 21305 raft_consensus.cc:2410] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 496a3c55c0994110b7824e8e0ea4015a for term 5 because replica has last-logged OpId of term: 3 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 17.
I20251024 20:01:29.527405 21396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "496a3c55c0994110b7824e8e0ea4015a" candidate_term: 5 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "df8fdb366ca94c638b9886b02d552ed7" is_pre_election: true
I20251024 20:01:29.527504 21396 raft_consensus.cc:2468] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 496a3c55c0994110b7824e8e0ea4015a in term 4.
I20251024 20:01:29.527652 20836 leader_election.cc:304] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 496a3c55c0994110b7824e8e0ea4015a, df8fdb366ca94c638b9886b02d552ed7; no voters: a339d74b39234a11a469995395989e7a
I20251024 20:01:29.527755 21350 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251024 20:01:29.527797 21350 raft_consensus.cc:493] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 20:01:29.527812 21350 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 4 FOLLOWER]: Advancing to term 5
I20251024 20:01:29.528420 21350 raft_consensus.cc:515] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.528540 21350 leader_election.cc:290] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [CANDIDATE]: Term 5 election: Requested vote from peers a339d74b39234a11a469995395989e7a (127.18.153.126:38517), df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773)
I20251024 20:01:29.528694 21396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "496a3c55c0994110b7824e8e0ea4015a" candidate_term: 5 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "df8fdb366ca94c638b9886b02d552ed7"
I20251024 20:01:29.528754 21396 raft_consensus.cc:3060] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 4 FOLLOWER]: Advancing to term 5
I20251024 20:01:29.529443 21396 raft_consensus.cc:2468] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 496a3c55c0994110b7824e8e0ea4015a in term 5.
I20251024 20:01:29.529532 21305 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "496a3c55c0994110b7824e8e0ea4015a" candidate_term: 5 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "a339d74b39234a11a469995395989e7a"
I20251024 20:01:29.529594 21305 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 4 FOLLOWER]: Advancing to term 5
I20251024 20:01:29.529613 20836 leader_election.cc:304] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 496a3c55c0994110b7824e8e0ea4015a, df8fdb366ca94c638b9886b02d552ed7; no voters: 
I20251024 20:01:29.529711 21350 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 5 FOLLOWER]: Leader election won for term 5
I20251024 20:01:29.529765 21350 raft_consensus.cc:697] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 5 LEADER]: Becoming Leader. State: Replica: 496a3c55c0994110b7824e8e0ea4015a, State: Running, Role: LEADER
I20251024 20:01:29.529816 21350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 2.17, Last appended by leader: 17, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.530021 21263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 496a3c55c0994110b7824e8e0ea4015a. Latest consensus state: current_term: 5 leader_uuid: "496a3c55c0994110b7824e8e0ea4015a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.530098 21263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:29.530233 21442 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:01:29.530372 21305 raft_consensus.cc:2410] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 496a3c55c0994110b7824e8e0ea4015a for term 5 because replica has last-logged OpId of term: 3 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 17.
I20251024 20:01:29.530467 21442 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300]
I20251024 20:01:29.530587 21442 tablet_loader.cc:96] loaded metadata for tablet 2dbef0e1505045468e973459c151f622 (table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300])
I20251024 20:01:29.530643 21442 tablet_loader.cc:96] loaded metadata for tablet 784c0490ffca4ccaa5cece7bae6064c2 (table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300])
I20251024 20:01:29.530689 21442 tablet_loader.cc:96] loaded metadata for tablet a4e6dc18db1e49208af2fa5dfaae96a6 (table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300])
I20251024 20:01:29.530726 21442 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:01:29.530795 21442 catalog_manager.cc:1269] Loaded cluster ID: bbb04eb8879d4aa8a0e7ff573f23a582
I20251024 20:01:29.530823 21442 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:01:29.530984 21442 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:01:29.531059 21442 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a: Loaded TSK: 0
I20251024 20:01:29.531148 21442 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:01:29.575744 21359 raft_consensus.cc:493] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 20:01:29.575840 21359 raft_consensus.cc:515] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.575986 21359 leader_election.cc:290] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773), 496a3c55c0994110b7824e8e0ea4015a (127.18.153.124:38423)
I20251024 20:01:29.576190 20862 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a339d74b39234a11a469995395989e7a" candidate_term: 6 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "496a3c55c0994110b7824e8e0ea4015a" is_pre_election: true
I20251024 20:01:29.578608 21396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a339d74b39234a11a469995395989e7a" candidate_term: 6 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "df8fdb366ca94c638b9886b02d552ed7" is_pre_election: true
I20251024 20:01:29.578688 21396 raft_consensus.cc:2468] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a339d74b39234a11a469995395989e7a in term 5.
I20251024 20:01:29.578824 21280 leader_election.cc:304] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a339d74b39234a11a469995395989e7a, df8fdb366ca94c638b9886b02d552ed7; no voters: 496a3c55c0994110b7824e8e0ea4015a
I20251024 20:01:29.578938 21359 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251024 20:01:29.578984 21359 raft_consensus.cc:493] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 20:01:29.579018 21359 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 5 FOLLOWER]: Advancing to term 6
I20251024 20:01:29.579699 21359 raft_consensus.cc:515] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.579836 21359 leader_election.cc:290] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [CANDIDATE]: Term 6 election: Requested vote from peers df8fdb366ca94c638b9886b02d552ed7 (127.18.153.125:38773), 496a3c55c0994110b7824e8e0ea4015a (127.18.153.124:38423)
I20251024 20:01:29.579980 21396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a339d74b39234a11a469995395989e7a" candidate_term: 6 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "df8fdb366ca94c638b9886b02d552ed7"
I20251024 20:01:29.579991 20862 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a339d74b39234a11a469995395989e7a" candidate_term: 6 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "496a3c55c0994110b7824e8e0ea4015a"
I20251024 20:01:29.580066 21396 raft_consensus.cc:3060] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 5 FOLLOWER]: Advancing to term 6
I20251024 20:01:29.580605 21396 raft_consensus.cc:2468] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a339d74b39234a11a469995395989e7a in term 6.
I20251024 20:01:29.580755 21280 leader_election.cc:304] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a339d74b39234a11a469995395989e7a, df8fdb366ca94c638b9886b02d552ed7; no voters: 496a3c55c0994110b7824e8e0ea4015a
I20251024 20:01:29.580844 21359 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 6 FOLLOWER]: Leader election won for term 6
I20251024 20:01:29.580924 21359 raft_consensus.cc:697] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 6 LEADER]: Becoming Leader. State: Replica: a339d74b39234a11a469995395989e7a, State: Running, Role: LEADER
I20251024 20:01:29.580976 21359 consensus_queue.cc:237] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 3.18, Last appended by leader: 18, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.581211 21334 sys_catalog.cc:455] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a339d74b39234a11a469995395989e7a. Latest consensus state: current_term: 6 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.581274 21334 sys_catalog.cc:458] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:29.581331 21360 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:01:29.581514 21360 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300]
I20251024 20:01:29.581600 21360 tablet_loader.cc:96] loaded metadata for tablet 2dbef0e1505045468e973459c151f622 (table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300])
I20251024 20:01:29.581650 21360 tablet_loader.cc:96] loaded metadata for tablet 784c0490ffca4ccaa5cece7bae6064c2 (table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300])
I20251024 20:01:29.581693 21360 tablet_loader.cc:96] loaded metadata for tablet a4e6dc18db1e49208af2fa5dfaae96a6 (table NewTableName [id=ef884b5c6c8b4ed6bc23772e0adeb300])
I20251024 20:01:29.581737 21360 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:01:29.581801 21360 catalog_manager.cc:1269] Loaded cluster ID: bbb04eb8879d4aa8a0e7ff573f23a582
I20251024 20:01:29.581831 21360 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:01:29.581986 21360 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:01:29.582054 21360 catalog_manager.cc:6033] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a: Loaded TSK: 0
I20251024 20:01:29.582150 21360 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:01:29.621930 21396 raft_consensus.cc:1240] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 6 FOLLOWER]: Rejecting Update request from peer 496a3c55c0994110b7824e8e0ea4015a for earlier term 5. Current term is 6. Ops: []
I20251024 20:01:29.622254 21263 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20251024 20:01:29.622409 21263 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 5 LEADER]: Stepping down as leader of term 5
I20251024 20:01:29.622445 21263 raft_consensus.cc:740] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 496a3c55c0994110b7824e8e0ea4015a, State: Running, Role: LEADER
I20251024 20:01:29.622511 21263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 5.18, Last appended by leader: 18, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } }
I20251024 20:01:29.622583 21263 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 5 FOLLOWER]: Advancing to term 6
I20251024 20:01:29.672248 21175 heartbeater.cc:499] Master 127.18.153.124:38423 was elected leader, sending a full tablet report...
I20251024 20:01:29.672266 21096 heartbeater.cc:499] Master 127.18.153.124:38423 was elected leader, sending a full tablet report...
I20251024 20:01:29.672327 21017 heartbeater.cc:499] Master 127.18.153.124:38423 was elected leader, sending a full tablet report...
I20251024 20:01:29.689955 20862 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 6 FOLLOWER]: Refusing update from remote peer a339d74b39234a11a469995395989e7a: Log matching property violated. Preceding OpId in replica: term: 5 index: 18. Preceding OpId from leader: term: 6 index: 19. (index mismatch)
I20251024 20:01:29.690318 21334 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20251024 20:01:29.690423 21334 consensus_queue.cc:1243] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [LEADER]: Peer 496a3c55c0994110b7824e8e0ea4015a log is divergent from this leader: its last log entry 5.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20251024 20:01:29.690659 20862 pending_rounds.cc:85] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a: Aborting all ops after (but not including) 17
I20251024 20:01:29.690717 20862 pending_rounds.cc:107] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a: Aborting uncommitted NO_OP operation due to leader change: 5.18
I20251024 20:01:29.690737 20862 raft_consensus.cc:2889] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251024 20:01:29.690886 21263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a339d74b39234a11a469995395989e7a. Latest consensus state: current_term: 6 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.690964 21263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:29.691334 21359 sys_catalog.cc:455] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.691438 21359 sys_catalog.cc:458] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:29.691571 21396 raft_consensus.cc:1275] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 6 FOLLOWER]: Refusing update from remote peer a339d74b39234a11a469995395989e7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 6 index: 19. (index mismatch)
I20251024 20:01:29.691597 21350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.691603 21263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.691699 21263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:29.691686 21350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:29.691807 21444 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20251024 20:01:29.692394 21425 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a339d74b39234a11a469995395989e7a. Latest consensus state: current_term: 6 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.692478 21425 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:29.692438 21448 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.692539 21448 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:29.692440 21449 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.692621 21449 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 20:01:29.772922 21018 heartbeater.cc:499] Master 127.18.153.126:38517 was elected leader, sending a full tablet report...
I20251024 20:01:29.772924 21176 heartbeater.cc:499] Master 127.18.153.126:38517 was elected leader, sending a full tablet report...
I20251024 20:01:29.773429 21097 heartbeater.cc:499] Master 127.18.153.126:38517 was elected leader, sending a full tablet report...
I20251024 20:01:29.800441 21359 sys_catalog.cc:455] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "a339d74b39234a11a469995395989e7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a339d74b39234a11a469995395989e7a" member_type: VOTER last_known_addr { host: "127.18.153.126" port: 38517 } } peers { permanent_uuid: "df8fdb366ca94c638b9886b02d552ed7" member_type: VOTER last_known_addr { host: "127.18.153.125" port: 38773 } } peers { permanent_uuid: "496a3c55c0994110b7824e8e0ea4015a" member_type: VOTER last_known_addr { host: "127.18.153.124" port: 38423 } } }
I20251024 20:01:29.800598 21359 sys_catalog.cc:458] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [sys.catalog]: This master's current role is: LEADER
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/ts_tablet_manager-itest.cc:786: Failure
Expected equality of these values:
  live_row_count
    Which is: 795
  table_info->GetMetrics()->live_row_count->value()
    Which is: 0
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/ts_tablet_manager-itest.cc:768: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
  Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/ts_tablet_manager-itest.cc:786: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 786, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 786, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
  Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/ts_tablet_manager-itest.cc:896: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20251024 20:01:59.603470 19045 tablet_server.cc:178] TabletServer@127.18.153.65:0 shutting down...
I20251024 20:01:59.607218 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:59.607338 19045 tablet_replica.cc:333] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679: stopping tablet replica
I20251024 20:01:59.607427 19045 raft_consensus.cc:2243] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 2 LEADER]: Raft consensus shutting down.
I20251024 20:01:59.607492 19045 raft_consensus.cc:2272] T a4e6dc18db1e49208af2fa5dfaae96a6 P 300038e5d9694754b6855f62e5e5e679 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.607657 19045 tablet_replica.cc:333] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679: stopping tablet replica
I20251024 20:01:59.607687 19045 raft_consensus.cc:2243] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 3 LEADER]: Raft consensus shutting down.
I20251024 20:01:59.607717 19045 raft_consensus.cc:2272] T 2dbef0e1505045468e973459c151f622 P 300038e5d9694754b6855f62e5e5e679 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.607818 19045 tablet_replica.cc:333] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679: stopping tablet replica
I20251024 20:01:59.607847 19045 raft_consensus.cc:2243] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 3 LEADER]: Raft consensus shutting down.
I20251024 20:01:59.607882 19045 raft_consensus.cc:2272] T 784c0490ffca4ccaa5cece7bae6064c2 P 300038e5d9694754b6855f62e5e5e679 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.619843 19045 tablet_server.cc:195] TabletServer@127.18.153.65:0 shutdown complete.
I20251024 20:01:59.620924 19045 tablet_server.cc:178] TabletServer@127.18.153.66:0 shutting down...
I20251024 20:01:59.624572 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:59.624722 19045 tablet_replica.cc:333] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182: stopping tablet replica
I20251024 20:01:59.624792 19045 raft_consensus.cc:2243] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.624856 19045 raft_consensus.cc:2272] T a4e6dc18db1e49208af2fa5dfaae96a6 P e0594f5433ae41248b6004bf70361182 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.625110 19045 tablet_replica.cc:333] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182: stopping tablet replica
I20251024 20:01:59.625154 19045 raft_consensus.cc:2243] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.625211 19045 raft_consensus.cc:2272] T 2dbef0e1505045468e973459c151f622 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.625335 19045 tablet_replica.cc:333] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182: stopping tablet replica
I20251024 20:01:59.625370 19045 raft_consensus.cc:2243] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.625409 19045 raft_consensus.cc:2272] T 784c0490ffca4ccaa5cece7bae6064c2 P e0594f5433ae41248b6004bf70361182 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.637120 19045 tablet_server.cc:195] TabletServer@127.18.153.66:0 shutdown complete.
I20251024 20:01:59.638175 19045 tablet_server.cc:178] TabletServer@127.18.153.67:0 shutting down...
I20251024 20:01:59.641719 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:01:59.641884 19045 tablet_replica.cc:333] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4: stopping tablet replica
I20251024 20:01:59.641952 19045 raft_consensus.cc:2243] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.642010 19045 raft_consensus.cc:2272] T a4e6dc18db1e49208af2fa5dfaae96a6 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.642186 19045 tablet_replica.cc:333] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4: stopping tablet replica
I20251024 20:01:59.642225 19045 raft_consensus.cc:2243] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.642266 19045 raft_consensus.cc:2272] T 2dbef0e1505045468e973459c151f622 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.642395 19045 tablet_replica.cc:333] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4: stopping tablet replica
I20251024 20:01:59.642429 19045 raft_consensus.cc:2243] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.642470 19045 raft_consensus.cc:2272] T 784c0490ffca4ccaa5cece7bae6064c2 P 8b22fe88c0914cdeb565ee21f8b66cd4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.654212 19045 tablet_server.cc:195] TabletServer@127.18.153.67:0 shutdown complete.
I20251024 20:01:59.655160 19045 master.cc:561] Master@127.18.153.126:38517 shutting down...
I20251024 20:01:59.658114 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 6 LEADER]: Raft consensus shutting down.
I20251024 20:01:59.658205 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P a339d74b39234a11a469995395989e7a [term 6 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.658229 19045 tablet_replica.cc:333] stopping tablet replica
I20251024 20:01:59.670795 19045 master.cc:583] Master@127.18.153.126:38517 shutdown complete.
I20251024 20:01:59.672053 19045 master.cc:561] Master@127.18.153.125:38773 shutting down...
I20251024 20:01:59.674906 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 6 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.674978 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P df8fdb366ca94c638b9886b02d552ed7 [term 6 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.675016 19045 tablet_replica.cc:333] stopping tablet replica
I20251024 20:01:59.688220 19045 master.cc:583] Master@127.18.153.125:38773 shutdown complete.
I20251024 20:01:59.689450 19045 master.cc:561] Master@127.18.153.124:38423 shutting down...
I20251024 20:01:59.692299 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 6 FOLLOWER]: Raft consensus shutting down.
I20251024 20:01:59.692397 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a [term 6 FOLLOWER]: Raft consensus is shut down!
I20251024 20:01:59.692437 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P 496a3c55c0994110b7824e8e0ea4015a: stopping tablet replica
I20251024 20:01:59.704779 19045 master.cc:583] Master@127.18.153.124:38423 shutdown complete.
I20251024 20:01:59.706573 19045 test_util.cc:183] -----------------------------------------------
I20251024 20:01:59.706645 19045 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1761336053687453-19045-0
[  FAILED  ] TsTabletManagerITest.TestTableStats (32343 ms)
[ RUN      ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20251024 20:01:59.707664 19045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.153.126:46669
I20251024 20:01:59.707796 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:59.708959 21479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:59.709044 21480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:59.709088 21482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:59.709162 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:59.709231 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:59.709262 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:59.709277 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336119709278 us; error 0 us; skew 500 ppm
I20251024 20:01:59.709806 19045 webserver.cc:492] Webserver started at http://127.18.153.126:36313/ using document root <none> and password file <none>
I20251024 20:01:59.709877 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:59.709918 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:59.709960 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:59.710268 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/master-0-root/instance:
uuid: "08711bae34e14848815712b4b4397798"
format_stamp: "Formatted at 2025-10-24 20:01:59 on dist-test-slave-375r"
I20251024 20:01:59.711297 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:59.711752 21487 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:59.711886 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:59.711930 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/master-0-root
uuid: "08711bae34e14848815712b4b4397798"
format_stamp: "Formatted at 2025-10-24 20:01:59 on dist-test-slave-375r"
I20251024 20:01:59.711975 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:59.726006 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:59.726230 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:59.728772 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.126:46669
I20251024 20:01:59.732530 21539 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.126:46669 every 8 connection(s)
I20251024 20:01:59.732705 21540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:59.733757 21540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798: Bootstrap starting.
I20251024 20:01:59.734043 21540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:59.734511 21540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798: No bootstrap required, opened a new log
I20251024 20:01:59.734660 21540 raft_consensus.cc:359] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08711bae34e14848815712b4b4397798" member_type: VOTER }
I20251024 20:01:59.734714 21540 raft_consensus.cc:385] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:59.734736 21540 raft_consensus.cc:740] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08711bae34e14848815712b4b4397798, State: Initialized, Role: FOLLOWER
I20251024 20:01:59.734790 21540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08711bae34e14848815712b4b4397798" member_type: VOTER }
I20251024 20:01:59.734848 21540 raft_consensus.cc:399] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:01:59.734879 21540 raft_consensus.cc:493] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:01:59.734918 21540 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:01:59.735468 21540 raft_consensus.cc:515] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08711bae34e14848815712b4b4397798" member_type: VOTER }
I20251024 20:01:59.735528 21540 leader_election.cc:304] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08711bae34e14848815712b4b4397798; no voters: 
I20251024 20:01:59.735636 21540 leader_election.cc:290] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:01:59.735688 21543 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:01:59.735788 21540 sys_catalog.cc:565] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:01:59.735816 21543 raft_consensus.cc:697] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [term 1 LEADER]: Becoming Leader. State: Replica: 08711bae34e14848815712b4b4397798, State: Running, Role: LEADER
I20251024 20:01:59.735867 21543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08711bae34e14848815712b4b4397798" member_type: VOTER }
I20251024 20:01:59.736057 21544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08711bae34e14848815712b4b4397798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08711bae34e14848815712b4b4397798" member_type: VOTER } }
I20251024 20:01:59.736058 21545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08711bae34e14848815712b4b4397798. Latest consensus state: current_term: 1 leader_uuid: "08711bae34e14848815712b4b4397798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08711bae34e14848815712b4b4397798" member_type: VOTER } }
I20251024 20:01:59.736145 21544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:59.736200 21545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [sys.catalog]: This master's current role is: LEADER
I20251024 20:01:59.736341 21548 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:01:59.736480 21548 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:01:59.737030 21548 catalog_manager.cc:1357] Generated new cluster ID: 4533cef680e442d4b61e9e7253b6cb10
I20251024 20:01:59.737076 21548 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:01:59.737152 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251024 20:01:59.747815 21548 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 20:01:59.748214 21548 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:01:59.753870 21548 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798: Generated new TSK 0
I20251024 20:01:59.753952 21548 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:01:59.768997 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:59.770334 21561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:59.770398 21562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:59.770517 19045 server_base.cc:1047] running on GCE node
W20251024 20:01:59.770397 21564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:59.770663 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:59.770696 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:59.770716 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336119770716 us; error 0 us; skew 500 ppm
I20251024 20:01:59.771355 19045 webserver.cc:492] Webserver started at http://127.18.153.65:38041/ using document root <none> and password file <none>
I20251024 20:01:59.771485 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:59.771535 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:59.771587 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:59.771884 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-0-root/instance:
uuid: "e3be00b83dac4ab8aae3b075b7402582"
format_stamp: "Formatted at 2025-10-24 20:01:59 on dist-test-slave-375r"
I20251024 20:01:59.772756 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:59.773190 21569 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:59.773322 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:59.773375 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "e3be00b83dac4ab8aae3b075b7402582"
format_stamp: "Formatted at 2025-10-24 20:01:59 on dist-test-slave-375r"
I20251024 20:01:59.773424 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:59.797369 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:59.797602 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:59.797771 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:59.797967 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:59.797994 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:59.798027 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:59.798053 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:59.800923 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:45985
I20251024 20:01:59.800969 21632 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:45985 every 8 connection(s)
I20251024 20:01:59.801313 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:59.802433 21637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:59.802448 21638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:59.802600 21640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:59.802774 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:59.802889 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:59.802917 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:59.802938 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336119802938 us; error 0 us; skew 500 ppm
I20251024 20:01:59.803613 19045 webserver.cc:492] Webserver started at http://127.18.153.66:39603/ using document root <none> and password file <none>
I20251024 20:01:59.803704 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:59.803740 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:59.803776 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:59.804041 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-1-root/instance:
uuid: "e53ae8f10052413686c912aab434952a"
format_stamp: "Formatted at 2025-10-24 20:01:59 on dist-test-slave-375r"
I20251024 20:01:59.805007 21633 heartbeater.cc:344] Connected to a master server at 127.18.153.126:46669
I20251024 20:01:59.805083 21633 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:59.805179 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:01:59.805203 21633 heartbeater.cc:507] Master 127.18.153.126:46669 requested a full tablet report, sending...
I20251024 20:01:59.805441 21504 ts_manager.cc:194] Registered new tserver with Master: e3be00b83dac4ab8aae3b075b7402582 (127.18.153.65:45985)
I20251024 20:01:59.805716 21645 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:59.805871 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.001s	sys 0.000s
I20251024 20:01:59.805928 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-1-root
uuid: "e53ae8f10052413686c912aab434952a"
format_stamp: "Formatted at 2025-10-24 20:01:59 on dist-test-slave-375r"
I20251024 20:01:59.805979 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:59.806128 21504 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50378
I20251024 20:01:59.827528 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:59.827749 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:59.827905 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:59.828116 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:59.828142 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:59.828176 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:59.828203 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:59.830924 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.66:33747
I20251024 20:01:59.830997 21708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.66:33747 every 8 connection(s)
I20251024 20:01:59.831344 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:01:59.832480 21712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:59.832574 21713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:01:59.832692 21715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:01:59.832796 19045 server_base.cc:1047] running on GCE node
I20251024 20:01:59.832865 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:01:59.832893 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:01:59.832917 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336119832918 us; error 0 us; skew 500 ppm
I20251024 20:01:59.833571 19045 webserver.cc:492] Webserver started at http://127.18.153.67:36031/ using document root <none> and password file <none>
I20251024 20:01:59.833666 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:01:59.833719 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:01:59.833775 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:01:59.834148 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-2-root/instance:
uuid: "e2231bba8e47412c9c0a1931dcb158ca"
format_stamp: "Formatted at 2025-10-24 20:01:59 on dist-test-slave-375r"
I20251024 20:01:59.834647 21709 heartbeater.cc:344] Connected to a master server at 127.18.153.126:46669
I20251024 20:01:59.834714 21709 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:59.834853 21709 heartbeater.cc:507] Master 127.18.153.126:46669 requested a full tablet report, sending...
I20251024 20:01:59.835088 21504 ts_manager.cc:194] Registered new tserver with Master: e53ae8f10052413686c912aab434952a (127.18.153.66:33747)
I20251024 20:01:59.835335 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.000s
I20251024 20:01:59.835559 21504 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50390
I20251024 20:01:59.835922 21720 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:59.836031 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:59.836082 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-2-root
uuid: "e2231bba8e47412c9c0a1931dcb158ca"
format_stamp: "Formatted at 2025-10-24 20:01:59 on dist-test-slave-375r"
I20251024 20:01:59.836130 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:01:59.841596 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:01:59.841784 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:01:59.841954 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:01:59.842130 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:01:59.842157 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:59.842190 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:01:59.842216 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:01:59.845415 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.67:35049
I20251024 20:01:59.845479 21783 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.67:35049 every 8 connection(s)
I20251024 20:01:59.848279 21784 heartbeater.cc:344] Connected to a master server at 127.18.153.126:46669
I20251024 20:01:59.848336 21784 heartbeater.cc:461] Registering TS with master...
I20251024 20:01:59.848434 21784 heartbeater.cc:507] Master 127.18.153.126:46669 requested a full tablet report, sending...
I20251024 20:01:59.848623 21504 ts_manager.cc:194] Registered new tserver with Master: e2231bba8e47412c9c0a1931dcb158ca (127.18.153.67:35049)
I20251024 20:01:59.848897 19045 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003289578s
I20251024 20:01:59.848995 21504 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50404
I20251024 20:01:59.855937 21504 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50420:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
  }
}
W20251024 20:01:59.856073 21504 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 20:01:59.860424 21748 tablet_service.cc:1505] Processing CreateTablet for tablet f8967fecd3f4433b86f31a784d125633 (DEFAULT_TABLE table=test-table [id=50b9f3fce7144906a7551254c4cd7803]), partition=
I20251024 20:01:59.860579 21748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8967fecd3f4433b86f31a784d125633. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:59.860922 21597 tablet_service.cc:1505] Processing CreateTablet for tablet f8967fecd3f4433b86f31a784d125633 (DEFAULT_TABLE table=test-table [id=50b9f3fce7144906a7551254c4cd7803]), partition=
I20251024 20:01:59.860980 21673 tablet_service.cc:1505] Processing CreateTablet for tablet f8967fecd3f4433b86f31a784d125633 (DEFAULT_TABLE table=test-table [id=50b9f3fce7144906a7551254c4cd7803]), partition=
I20251024 20:01:59.861020 21597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8967fecd3f4433b86f31a784d125633. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:59.861075 21673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8967fecd3f4433b86f31a784d125633. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:01:59.862082 21800 tablet_bootstrap.cc:492] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582: Bootstrap starting.
I20251024 20:01:59.862294 21801 tablet_bootstrap.cc:492] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca: Bootstrap starting.
I20251024 20:01:59.862509 21800 tablet_bootstrap.cc:654] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:59.862773 21801 tablet_bootstrap.cc:654] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:59.862802 21802 tablet_bootstrap.cc:492] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Bootstrap starting.
I20251024 20:01:59.863171 21802 tablet_bootstrap.cc:654] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Neither blocks nor log segments found. Creating new log.
I20251024 20:01:59.863338 21800 tablet_bootstrap.cc:492] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582: No bootstrap required, opened a new log
I20251024 20:01:59.863407 21800 ts_tablet_manager.cc:1403] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:59.863422 21801 tablet_bootstrap.cc:492] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca: No bootstrap required, opened a new log
I20251024 20:01:59.863463 21801 ts_tablet_manager.cc:1403] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:59.863576 21800 raft_consensus.cc:359] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2231bba8e47412c9c0a1931dcb158ca" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 35049 } } peers { permanent_uuid: "e53ae8f10052413686c912aab434952a" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 33747 } } peers { permanent_uuid: "e3be00b83dac4ab8aae3b075b7402582" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45985 } }
I20251024 20:01:59.863629 21800 raft_consensus.cc:385] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:59.863644 21800 raft_consensus.cc:740] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3be00b83dac4ab8aae3b075b7402582, State: Initialized, Role: FOLLOWER
I20251024 20:01:59.863600 21801 raft_consensus.cc:359] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2231bba8e47412c9c0a1931dcb158ca" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 35049 } } peers { permanent_uuid: "e53ae8f10052413686c912aab434952a" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 33747 } } peers { permanent_uuid: "e3be00b83dac4ab8aae3b075b7402582" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45985 } }
I20251024 20:01:59.863672 21801 raft_consensus.cc:385] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:59.863694 21800 consensus_queue.cc:260] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2231bba8e47412c9c0a1931dcb158ca" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 35049 } } peers { permanent_uuid: "e53ae8f10052413686c912aab434952a" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 33747 } } peers { permanent_uuid: "e3be00b83dac4ab8aae3b075b7402582" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45985 } }
I20251024 20:01:59.863696 21801 raft_consensus.cc:740] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2231bba8e47412c9c0a1931dcb158ca, State: Initialized, Role: FOLLOWER
I20251024 20:01:59.863796 21802 tablet_bootstrap.cc:492] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: No bootstrap required, opened a new log
I20251024 20:01:59.863845 21802 ts_tablet_manager.cc:1403] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:59.863802 21801 consensus_queue.cc:260] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2231bba8e47412c9c0a1931dcb158ca" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 35049 } } peers { permanent_uuid: "e53ae8f10052413686c912aab434952a" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 33747 } } peers { permanent_uuid: "e3be00b83dac4ab8aae3b075b7402582" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45985 } }
I20251024 20:01:59.863880 21800 ts_tablet_manager.cc:1434] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582: Time spent starting tablet: real 0.000s	user 0.000s	sys 0.000s
W20251024 20:01:59.863926 21800 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20251024 20:01:59.863993 21802 raft_consensus.cc:359] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2231bba8e47412c9c0a1931dcb158ca" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 35049 } } peers { permanent_uuid: "e53ae8f10052413686c912aab434952a" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 33747 } } peers { permanent_uuid: "e3be00b83dac4ab8aae3b075b7402582" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45985 } }
I20251024 20:01:59.864019 21801 ts_tablet_manager.cc:1434] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:01:59.864049 21802 raft_consensus.cc:385] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:01:59.864054 21633 heartbeater.cc:499] Master 127.18.153.126:46669 was elected leader, sending a full tablet report...
I20251024 20:01:59.864080 21802 raft_consensus.cc:740] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53ae8f10052413686c912aab434952a, State: Initialized, Role: FOLLOWER
W20251024 20:01:59.864053 21801 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20251024 20:01:59.864164 21802 consensus_queue.cc:260] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2231bba8e47412c9c0a1931dcb158ca" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 35049 } } peers { permanent_uuid: "e53ae8f10052413686c912aab434952a" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 33747 } } peers { permanent_uuid: "e3be00b83dac4ab8aae3b075b7402582" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45985 } }
I20251024 20:01:59.864300 21802 ts_tablet_manager.cc:1434] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Time spent starting tablet: real 0.000s	user 0.001s	sys 0.000s
I20251024 20:01:59.864317 21784 heartbeater.cc:499] Master 127.18.153.126:46669 was elected leader, sending a full tablet report...
W20251024 20:01:59.864336 21802 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20251024 20:01:59.864430 21709 heartbeater.cc:499] Master 127.18.153.126:46669 was elected leader, sending a full tablet report...
I20251024 20:02:00.002210 21806 raft_consensus.cc:493] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 20:02:00.002346 21806 raft_consensus.cc:515] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2231bba8e47412c9c0a1931dcb158ca" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 35049 } } peers { permanent_uuid: "e53ae8f10052413686c912aab434952a" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 33747 } } peers { permanent_uuid: "e3be00b83dac4ab8aae3b075b7402582" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45985 } }
I20251024 20:02:00.002589 21806 leader_election.cc:290] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2231bba8e47412c9c0a1931dcb158ca (127.18.153.67:35049), e53ae8f10052413686c912aab434952a (127.18.153.66:33747)
I20251024 20:02:00.005231 21758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8967fecd3f4433b86f31a784d125633" candidate_uuid: "e3be00b83dac4ab8aae3b075b7402582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2231bba8e47412c9c0a1931dcb158ca" is_pre_election: true
I20251024 20:02:00.005234 21683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8967fecd3f4433b86f31a784d125633" candidate_uuid: "e3be00b83dac4ab8aae3b075b7402582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53ae8f10052413686c912aab434952a" is_pre_election: true
I20251024 20:02:00.005347 21758 raft_consensus.cc:2468] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3be00b83dac4ab8aae3b075b7402582 in term 0.
I20251024 20:02:00.005368 21683 raft_consensus.cc:2468] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3be00b83dac4ab8aae3b075b7402582 in term 0.
I20251024 20:02:00.005501 21571 leader_election.cc:304] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2231bba8e47412c9c0a1931dcb158ca, e3be00b83dac4ab8aae3b075b7402582; no voters: 
I20251024 20:02:00.005605 21806 raft_consensus.cc:2804] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 20:02:00.005654 21806 raft_consensus.cc:493] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 20:02:00.005674 21806 raft_consensus.cc:3060] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:00.006332 21806 raft_consensus.cc:515] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2231bba8e47412c9c0a1931dcb158ca" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 35049 } } peers { permanent_uuid: "e53ae8f10052413686c912aab434952a" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 33747 } } peers { permanent_uuid: "e3be00b83dac4ab8aae3b075b7402582" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45985 } }
I20251024 20:02:00.006453 21806 leader_election.cc:290] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [CANDIDATE]: Term 1 election: Requested vote from peers e2231bba8e47412c9c0a1931dcb158ca (127.18.153.67:35049), e53ae8f10052413686c912aab434952a (127.18.153.66:33747)
I20251024 20:02:00.006579 21683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8967fecd3f4433b86f31a784d125633" candidate_uuid: "e3be00b83dac4ab8aae3b075b7402582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53ae8f10052413686c912aab434952a"
I20251024 20:02:00.006587 21758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f8967fecd3f4433b86f31a784d125633" candidate_uuid: "e3be00b83dac4ab8aae3b075b7402582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2231bba8e47412c9c0a1931dcb158ca"
I20251024 20:02:00.006641 21683 raft_consensus.cc:3060] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:00.006647 21758 raft_consensus.cc:3060] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:00.007319 21683 raft_consensus.cc:2468] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3be00b83dac4ab8aae3b075b7402582 in term 1.
I20251024 20:02:00.007354 21758 raft_consensus.cc:2468] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3be00b83dac4ab8aae3b075b7402582 in term 1.
I20251024 20:02:00.007530 21570 leader_election.cc:304] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3be00b83dac4ab8aae3b075b7402582, e53ae8f10052413686c912aab434952a; no voters: 
I20251024 20:02:00.007619 21806 raft_consensus.cc:2804] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:02:00.007776 21806 raft_consensus.cc:697] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [term 1 LEADER]: Becoming Leader. State: Replica: e3be00b83dac4ab8aae3b075b7402582, State: Running, Role: LEADER
I20251024 20:02:00.007822 21806 consensus_queue.cc:237] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2231bba8e47412c9c0a1931dcb158ca" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 35049 } } peers { permanent_uuid: "e53ae8f10052413686c912aab434952a" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 33747 } } peers { permanent_uuid: "e3be00b83dac4ab8aae3b075b7402582" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45985 } }
I20251024 20:02:00.008353 21502 catalog_manager.cc:5649] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3be00b83dac4ab8aae3b075b7402582 (127.18.153.65). New cstate: current_term: 1 leader_uuid: "e3be00b83dac4ab8aae3b075b7402582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2231bba8e47412c9c0a1931dcb158ca" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e53ae8f10052413686c912aab434952a" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 33747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3be00b83dac4ab8aae3b075b7402582" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45985 } health_report { overall_health: HEALTHY } } }
I20251024 20:02:00.048022 21673 tablet_service.cc:1552] Processing DeleteTablet for tablet f8967fecd3f4433b86f31a784d125633 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:60396
I20251024 20:02:00.048293 21813 tablet_replica.cc:333] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: stopping tablet replica
I20251024 20:02:00.048369 21813 raft_consensus.cc:2243] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 20:02:00.048427 21813 raft_consensus.cc:2272] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:00.048672 21813 ts_tablet_manager.cc:1916] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251024 20:02:00.050375 21813 ts_tablet_manager.cc:1929] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 0.0
I20251024 20:02:00.050457 21813 log.cc:1199] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Deleting WAL directory at /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-1-root/wals/f8967fecd3f4433b86f31a784d125633
I20251024 20:02:00.051158 21814 ts_tablet_manager.cc:933] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Initiating tablet copy from peer e3be00b83dac4ab8aae3b075b7402582 (127.18.153.65:45985)
W20251024 20:02:00.051340 21634 tablet.cc:2378] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 20:02:00.051350 21814 tablet_copy_client.cc:323] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: tablet copy: Beginning tablet copy session from remote peer at address 127.18.153.65:45985
I20251024 20:02:00.053548 21617 tablet_copy_service.cc:140] P e3be00b83dac4ab8aae3b075b7402582: Received BeginTabletCopySession request for tablet f8967fecd3f4433b86f31a784d125633 from peer e53ae8f10052413686c912aab434952a ({username='slave'} at 127.0.0.1:38022)
I20251024 20:02:00.053603 21617 tablet_copy_service.cc:161] P e3be00b83dac4ab8aae3b075b7402582: Beginning new tablet copy session on tablet f8967fecd3f4433b86f31a784d125633 from peer e53ae8f10052413686c912aab434952a at {username='slave'} at 127.0.0.1:38022: session id = e53ae8f10052413686c912aab434952a-f8967fecd3f4433b86f31a784d125633
I20251024 20:02:00.054039 21617 tablet_copy_source_session.cc:215] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582: Tablet Copy: opened 0 blocks and 1 log segments
I20251024 20:02:00.054378 21814 ts_tablet_manager.cc:1916] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20251024 20:02:00.055634 21814 ts_tablet_manager.cc:1929] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 0.0
I20251024 20:02:00.055719 21814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8967fecd3f4433b86f31a784d125633. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:00.056214 21814 tablet_copy_client.cc:806] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: tablet copy: Starting download of 0 data blocks...
I20251024 20:02:00.056324 21814 tablet_copy_client.cc:670] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: tablet copy: Starting download of 1 WAL segments...
I20251024 20:02:00.056573 21502 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50434:
table { table_id: "50b9f3fce7144906a7551254c4cd7803" table_name: "test-table" }
I20251024 20:02:00.056643 21502 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50434:
table { table_id: "50b9f3fce7144906a7551254c4cd7803" table_name: "test-table" }
I20251024 20:02:00.056923 21814 tablet_copy_client.cc:538] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251024 20:02:00.057317 21502 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798: Sending DeleteTablet for 3 replicas of tablet f8967fecd3f4433b86f31a784d125633
I20251024 20:02:00.057595 21814 tablet_bootstrap.cc:492] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Bootstrap starting.
I20251024 20:02:00.057650 21673 tablet_service.cc:1552] Processing DeleteTablet for tablet f8967fecd3f4433b86f31a784d125633 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 20:02:00 UTC) from {username='slave'} at 127.0.0.1:60378
I20251024 20:02:00.057652 21597 tablet_service.cc:1552] Processing DeleteTablet for tablet f8967fecd3f4433b86f31a784d125633 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 20:02:00 UTC) from {username='slave'} at 127.0.0.1:37998
I20251024 20:02:00.057664 21748 tablet_service.cc:1552] Processing DeleteTablet for tablet f8967fecd3f4433b86f31a784d125633 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 20:02:00 UTC) from {username='slave'} at 127.0.0.1:37204
I20251024 20:02:00.057919 21816 tablet_replica.cc:333] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582: stopping tablet replica
I20251024 20:02:00.057994 21816 raft_consensus.cc:2243] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:00.058058 21816 pending_rounds.cc:70] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582: Trying to abort 1 pending ops.
I20251024 20:02:00.058086 21816 pending_rounds.cc:77] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214432747552550912 op_type: NO_OP noop_request { }
W20251024 20:02:00.058141 21488 catalog_manager.cc:4965] TS e53ae8f10052413686c912aab434952a (127.18.153.66:33747): delete failed for tablet f8967fecd3f4433b86f31a784d125633 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8967fecd3f4433b86f31a784d125633 already in progress: copying tablet
I20251024 20:02:00.058159 21816 raft_consensus.cc:2889] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 20:02:00.058202 21816 raft_consensus.cc:2272] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:00.058279 21817 tablet_replica.cc:333] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca: stopping tablet replica
I20251024 20:02:00.058331 21817 raft_consensus.cc:2243] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 20:02:00.058372 21817 raft_consensus.cc:2272] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:00.058522 21816 ts_tablet_manager.cc:1916] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 20:02:00.058563 21817 ts_tablet_manager.cc:1916] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 20:02:00.059052 21814 tablet_bootstrap.cc:492] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251024 20:02:00.059247 21814 tablet_bootstrap.cc:492] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Bootstrap complete.
I20251024 20:02:00.059355 21814 ts_tablet_manager.cc:1403] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20251024 20:02:00.059556 21814 raft_consensus.cc:359] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2231bba8e47412c9c0a1931dcb158ca" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 35049 } } peers { permanent_uuid: "e53ae8f10052413686c912aab434952a" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 33747 } } peers { permanent_uuid: "e3be00b83dac4ab8aae3b075b7402582" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45985 } }
I20251024 20:02:00.059645 21814 raft_consensus.cc:740] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53ae8f10052413686c912aab434952a, State: Initialized, Role: FOLLOWER
I20251024 20:02:00.059751 21814 consensus_queue.cc:260] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2231bba8e47412c9c0a1931dcb158ca" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 35049 } } peers { permanent_uuid: "e53ae8f10052413686c912aab434952a" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 33747 } } peers { permanent_uuid: "e3be00b83dac4ab8aae3b075b7402582" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 45985 } }
I20251024 20:02:00.059895 21816 ts_tablet_manager.cc:1929] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 20:02:00.059949 21816 log.cc:1199] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582: Deleting WAL directory at /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-0-root/wals/f8967fecd3f4433b86f31a784d125633
I20251024 20:02:00.060153 21816 ts_tablet_manager.cc:1950] T f8967fecd3f4433b86f31a784d125633 P e3be00b83dac4ab8aae3b075b7402582: Deleting consensus metadata
I20251024 20:02:00.060236 21817 ts_tablet_manager.cc:1929] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 20:02:00.060297 21673 tablet_service.cc:1552] Processing DeleteTablet for tablet f8967fecd3f4433b86f31a784d125633 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 20:02:00 UTC) from {username='slave'} at 127.0.0.1:60378
I20251024 20:02:00.060302 21817 log.cc:1199] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca: Deleting WAL directory at /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-2-root/wals/f8967fecd3f4433b86f31a784d125633
W20251024 20:02:00.060493 21488 catalog_manager.cc:4965] TS e53ae8f10052413686c912aab434952a (127.18.153.66:33747): delete failed for tablet f8967fecd3f4433b86f31a784d125633 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8967fecd3f4433b86f31a784d125633 already in progress: copying tablet
I20251024 20:02:00.060555 21489 catalog_manager.cc:4985] TS e3be00b83dac4ab8aae3b075b7402582 (127.18.153.65:45985): tablet f8967fecd3f4433b86f31a784d125633 (table test-table [id=50b9f3fce7144906a7551254c4cd7803]) successfully deleted
I20251024 20:02:00.060563 21814 ts_tablet_manager.cc:1434] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20251024 20:02:00.060623 21817 ts_tablet_manager.cc:1950] T f8967fecd3f4433b86f31a784d125633 P e2231bba8e47412c9c0a1931dcb158ca: Deleting consensus metadata
W20251024 20:02:00.060624 21814 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20251024 20:02:00.060915 21489 catalog_manager.cc:4985] TS e2231bba8e47412c9c0a1931dcb158ca (127.18.153.67:35049): tablet f8967fecd3f4433b86f31a784d125633 (table test-table [id=50b9f3fce7144906a7551254c4cd7803]) successfully deleted
I20251024 20:02:00.111132 21617 tablet_copy_service.cc:342] P e3be00b83dac4ab8aae3b075b7402582: Request end of tablet copy session e53ae8f10052413686c912aab434952a-f8967fecd3f4433b86f31a784d125633 received from {username='slave'} at 127.0.0.1:38022
I20251024 20:02:00.111245 21617 tablet_copy_service.cc:434] P e3be00b83dac4ab8aae3b075b7402582: ending tablet copy session e53ae8f10052413686c912aab434952a-f8967fecd3f4433b86f31a784d125633 on tablet f8967fecd3f4433b86f31a784d125633 with peer e53ae8f10052413686c912aab434952a
I20251024 20:02:00.111421 21673 tablet_service.cc:1552] Processing DeleteTablet for tablet f8967fecd3f4433b86f31a784d125633 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 20:02:00 UTC) from {username='slave'} at 127.0.0.1:60378
I20251024 20:02:00.111557 21813 tablet_replica.cc:333] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: stopping tablet replica
I20251024 20:02:00.111625 21813 raft_consensus.cc:2243] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 20:02:00.111675 21813 pending_rounds.cc:70] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Trying to abort 1 pending ops.
I20251024 20:02:00.111696 21813 pending_rounds.cc:77] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214432747552550912 op_type: NO_OP noop_request { }
I20251024 20:02:00.111755 21813 raft_consensus.cc:2889] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 20:02:00.111789 21813 raft_consensus.cc:2272] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:00.111994 21813 ts_tablet_manager.cc:1916] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 20:02:00.113620 21813 ts_tablet_manager.cc:1929] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 20:02:00.113684 21813 log.cc:1199] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Deleting WAL directory at /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1761336053687453-19045-0/minicluster-data/ts-1-root/wals/f8967fecd3f4433b86f31a784d125633
I20251024 20:02:00.113878 21813 ts_tablet_manager.cc:1950] T f8967fecd3f4433b86f31a784d125633 P e53ae8f10052413686c912aab434952a: Deleting consensus metadata
I20251024 20:02:00.114171 21488 catalog_manager.cc:4985] TS e53ae8f10052413686c912aab434952a (127.18.153.66:33747): tablet f8967fecd3f4433b86f31a784d125633 (table test-table [id=50b9f3fce7144906a7551254c4cd7803]) successfully deleted
I20251024 20:02:00.122274 19045 tablet_server.cc:178] TabletServer@127.18.153.65:0 shutting down...
I20251024 20:02:00.124814 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:00.136724 19045 tablet_server.cc:195] TabletServer@127.18.153.65:0 shutdown complete.
I20251024 20:02:00.137810 19045 tablet_server.cc:178] TabletServer@127.18.153.66:0 shutting down...
I20251024 20:02:00.139914 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:00.151842 19045 tablet_server.cc:195] TabletServer@127.18.153.66:0 shutdown complete.
I20251024 20:02:00.152740 19045 tablet_server.cc:178] TabletServer@127.18.153.67:0 shutting down...
I20251024 20:02:00.154960 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:00.166596 19045 tablet_server.cc:195] TabletServer@127.18.153.67:0 shutdown complete.
I20251024 20:02:00.167572 19045 master.cc:561] Master@127.18.153.126:46669 shutting down...
I20251024 20:02:00.169975 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:00.170047 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:00.170140 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P 08711bae34e14848815712b4b4397798: stopping tablet replica
I20251024 20:02:00.182057 19045 master.cc:583] Master@127.18.153.126:46669 shutdown complete.
[       OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (478 ms)
[----------] 4 tests from TsTabletManagerITest (35058 ms total)

[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN      ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20251024 20:02:00.186242 19045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.153.126:36441
I20251024 20:02:00.186393 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:00.187444 21823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:00.187487 21826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:00.187618 21824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.187672 19045 server_base.cc:1047] running on GCE node
I20251024 20:02:00.187764 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:00.187793 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:00.187805 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336120187806 us; error 0 us; skew 500 ppm
I20251024 20:02:00.188373 19045 webserver.cc:492] Webserver started at http://127.18.153.126:42385/ using document root <none> and password file <none>
I20251024 20:02:00.188460 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:00.188510 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:00.188561 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:00.188827 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/master-0-root/instance:
uuid: "d8b0403f78c944c5825978d06404a74a"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.189697 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:00.190105 21831 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.190213 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.190255 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/master-0-root
uuid: "d8b0403f78c944c5825978d06404a74a"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.190303 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:00.211014 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:00.211225 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:00.213781 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.126:36441
I20251024 20:02:00.216058 21883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.126:36441 every 8 connection(s)
I20251024 20:02:00.216187 21884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:00.217175 21884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a: Bootstrap starting.
I20251024 20:02:00.217464 21884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:00.217945 21884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a: No bootstrap required, opened a new log
I20251024 20:02:00.218062 21884 raft_consensus.cc:359] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b0403f78c944c5825978d06404a74a" member_type: VOTER }
I20251024 20:02:00.218122 21884 raft_consensus.cc:385] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:00.218137 21884 raft_consensus.cc:740] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b0403f78c944c5825978d06404a74a, State: Initialized, Role: FOLLOWER
I20251024 20:02:00.218186 21884 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b0403f78c944c5825978d06404a74a" member_type: VOTER }
I20251024 20:02:00.218225 21884 raft_consensus.cc:399] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:02:00.218255 21884 raft_consensus.cc:493] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:02:00.218284 21884 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:00.218767 21884 raft_consensus.cc:515] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b0403f78c944c5825978d06404a74a" member_type: VOTER }
I20251024 20:02:00.218825 21884 leader_election.cc:304] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8b0403f78c944c5825978d06404a74a; no voters: 
I20251024 20:02:00.218924 21884 leader_election.cc:290] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:02:00.218992 21887 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:02:00.219060 21884 sys_catalog.cc:565] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:02:00.219136 21887 raft_consensus.cc:697] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [term 1 LEADER]: Becoming Leader. State: Replica: d8b0403f78c944c5825978d06404a74a, State: Running, Role: LEADER
I20251024 20:02:00.219184 21887 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b0403f78c944c5825978d06404a74a" member_type: VOTER }
I20251024 20:02:00.219350 21887 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8b0403f78c944c5825978d06404a74a. Latest consensus state: current_term: 1 leader_uuid: "d8b0403f78c944c5825978d06404a74a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b0403f78c944c5825978d06404a74a" member_type: VOTER } }
I20251024 20:02:00.219431 21887 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [sys.catalog]: This master's current role is: LEADER
I20251024 20:02:00.219394 21889 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8b0403f78c944c5825978d06404a74a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b0403f78c944c5825978d06404a74a" member_type: VOTER } }
I20251024 20:02:00.219610 21889 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [sys.catalog]: This master's current role is: LEADER
I20251024 20:02:00.219724 21892 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:02:00.219857 21892 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:02:00.220431 21892 catalog_manager.cc:1357] Generated new cluster ID: 5da2471ad6eb47a6bb7a3725e95384ed
I20251024 20:02:00.220469 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251024 20:02:00.220474 21892 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:02:00.226538 21892 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 20:02:00.226975 21892 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:02:00.230508 21892 catalog_manager.cc:6022] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a: Generated new TSK 0
I20251024 20:02:00.230587 21892 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:02:00.236229 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251024 20:02:00.237605 19045 server_base.cc:1047] running on GCE node
W20251024 20:02:00.237686 21908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:00.237754 21906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:00.237701 21905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.238000 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:00.238035 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:00.238065 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336120238066 us; error 0 us; skew 500 ppm
I20251024 20:02:00.238620 19045 webserver.cc:492] Webserver started at http://127.18.153.65:38211/ using document root <none> and password file <none>
I20251024 20:02:00.238690 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:00.238724 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:00.238760 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:00.239012 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root/instance:
uuid: "356eb15ea83947ae821c9f0ad8fc5457"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.239948 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:02:00.240384 21913 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.240496 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.240538 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "356eb15ea83947ae821c9f0ad8fc5457"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.240585 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:00.252110 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:00.252341 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:00.252679 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:02:00.252710 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.252744 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:02:00.252771 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.255941 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:43339
I20251024 20:02:00.255987 21979 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:43339 every 8 connection(s)
I20251024 20:02:00.256750 21980 heartbeater.cc:344] Connected to a master server at 127.18.153.126:36441
I20251024 20:02:00.256810 21980 heartbeater.cc:461] Registering TS with master...
I20251024 20:02:00.256907 21980 heartbeater.cc:507] Master 127.18.153.126:36441 requested a full tablet report, sending...
I20251024 20:02:00.257154 21847 ts_manager.cc:194] Registered new tserver with Master: 356eb15ea83947ae821c9f0ad8fc5457 (127.18.153.65:43339)
I20251024 20:02:00.257272 19045 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.001113912s
I20251024 20:02:00.257984 21847 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59258
I20251024 20:02:00.264808 21944 tablet_service.cc:1505] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20251024 20:02:00.264959 21944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:00.266108 21996 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: Bootstrap starting.
I20251024 20:02:00.266505 21996 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:00.266983 21996 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: No bootstrap required, opened a new log
I20251024 20:02:00.267019 21996 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:00.267158 21996 raft_consensus.cc:359] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "356eb15ea83947ae821c9f0ad8fc5457" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 43339 } }
I20251024 20:02:00.267207 21996 raft_consensus.cc:385] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:00.267225 21996 raft_consensus.cc:740] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356eb15ea83947ae821c9f0ad8fc5457, State: Initialized, Role: FOLLOWER
I20251024 20:02:00.267271 21996 consensus_queue.cc:260] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "356eb15ea83947ae821c9f0ad8fc5457" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 43339 } }
I20251024 20:02:00.267315 21996 raft_consensus.cc:399] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:02:00.267335 21996 raft_consensus.cc:493] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:02:00.267356 21996 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:00.267855 21996 raft_consensus.cc:515] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "356eb15ea83947ae821c9f0ad8fc5457" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 43339 } }
I20251024 20:02:00.267910 21996 leader_election.cc:304] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 356eb15ea83947ae821c9f0ad8fc5457; no voters: 
I20251024 20:02:00.268008 21996 leader_election.cc:290] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:02:00.268072 21998 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:02:00.268134 21996 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:02:00.268168 21980 heartbeater.cc:499] Master 127.18.153.126:36441 was elected leader, sending a full tablet report...
I20251024 20:02:00.268222 21998 raft_consensus.cc:697] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 1 LEADER]: Becoming Leader. State: Replica: 356eb15ea83947ae821c9f0ad8fc5457, State: Running, Role: LEADER
I20251024 20:02:00.268282 21998 consensus_queue.cc:237] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "356eb15ea83947ae821c9f0ad8fc5457" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 43339 } }
I20251024 20:02:00.268528 22000 tablet_replica.cc:442] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: TxnStatusTablet state changed. Reason: New leader 356eb15ea83947ae821c9f0ad8fc5457. Latest consensus state: current_term: 1 leader_uuid: "356eb15ea83947ae821c9f0ad8fc5457" committed_config { opid_index: -1 peers { permanent_uuid: "356eb15ea83947ae821c9f0ad8fc5457" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 43339 } } }
I20251024 20:02:00.268548 21999 tablet_replica.cc:442] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "356eb15ea83947ae821c9f0ad8fc5457" committed_config { opid_index: -1 peers { permanent_uuid: "356eb15ea83947ae821c9f0ad8fc5457" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 43339 } } }
I20251024 20:02:00.268620 22000 tablet_replica.cc:445] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: This TxnStatusTablet replica's current role is: LEADER
I20251024 20:02:00.268626 21999 tablet_replica.cc:445] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: This TxnStatusTablet replica's current role is: LEADER
W20251024 20:02:00.268692 21847 catalog_manager.cc:5482] Ignoring report from unknown tablet 11111111111111111111111111111111
I20251024 20:02:00.268823 22002 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251024 20:02:00.268900 22002 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20251024 20:02:00.269029 21847 catalog_manager.cc:5482] Ignoring report from unknown tablet 11111111111111111111111111111111
I20251024 20:02:00.270599 19045 tablet_server.cc:178] TabletServer@127.18.153.65:0 shutting down...
I20251024 20:02:00.272972 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:00.273092 19045 tablet_replica.cc:333] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: stopping tablet replica
I20251024 20:02:00.273155 19045 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:00.273211 19045 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:00.285327 19045 tablet_server.cc:195] TabletServer@127.18.153.65:0 shutdown complete.
I20251024 20:02:00.286854 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:00.288023 22006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:00.288028 22009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:00.288079 22007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.288304 19045 server_base.cc:1047] running on GCE node
I20251024 20:02:00.288400 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:00.288450 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:00.288466 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336120288466 us; error 0 us; skew 500 ppm
I20251024 20:02:00.289085 19045 webserver.cc:492] Webserver started at http://127.18.153.65:38211/ using document root <none> and password file <none>
I20251024 20:02:00.289197 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:00.289227 19045 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251024 20:02:00.289770 19045 fs_manager.cc:714] Time spent opening directory manager: real 0.000s	user 0.001s	sys 0.000s
I20251024 20:02:00.290243 22014 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.290402 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.290447 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "356eb15ea83947ae821c9f0ad8fc5457"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.290503 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:00.308779 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:00.308979 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:00.309645 22022 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251024 20:02:00.310415 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251024 20:02:00.310467 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
I20251024 20:02:00.310500 19045 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251024 20:02:00.310983 19045 ts_tablet_manager.cc:616] Registered 1 tablets
I20251024 20:02:00.311028 22022 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: Bootstrap starting.
I20251024 20:02:00.311036 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
I20251024 20:02:00.313604 22022 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251024 20:02:00.313871 22022 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: Bootstrap complete.
I20251024 20:02:00.313978 22022 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20251024 20:02:00.314132 22022 raft_consensus.cc:359] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "356eb15ea83947ae821c9f0ad8fc5457" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 43339 } }
I20251024 20:02:00.314258 22022 raft_consensus.cc:740] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356eb15ea83947ae821c9f0ad8fc5457, State: Initialized, Role: FOLLOWER
I20251024 20:02:00.314323 22022 consensus_queue.cc:260] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "356eb15ea83947ae821c9f0ad8fc5457" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 43339 } }
I20251024 20:02:00.314373 22022 raft_consensus.cc:399] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:02:00.314399 22022 raft_consensus.cc:493] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:02:00.314419 22022 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:02:00.315341 22022 raft_consensus.cc:515] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "356eb15ea83947ae821c9f0ad8fc5457" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 43339 } }
I20251024 20:02:00.315454 22022 leader_election.cc:304] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 356eb15ea83947ae821c9f0ad8fc5457; no voters: 
I20251024 20:02:00.315503 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:43339
I20251024 20:02:00.315557 22022 leader_election.cc:290] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [CANDIDATE]: Term 2 election: Requested vote from peers 
I20251024 20:02:00.315716 22088 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 2 FOLLOWER]: Leader election won for term 2
I20251024 20:02:00.315723 22022 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20251024 20:02:00.315888 22088 raft_consensus.cc:697] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 2 LEADER]: Becoming Leader. State: Replica: 356eb15ea83947ae821c9f0ad8fc5457, State: Running, Role: LEADER
I20251024 20:02:00.315944 22088 consensus_queue.cc:237] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "356eb15ea83947ae821c9f0ad8fc5457" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 43339 } }
I20251024 20:02:00.316170 22088 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 356eb15ea83947ae821c9f0ad8fc5457. Latest consensus state: current_term: 2 leader_uuid: "356eb15ea83947ae821c9f0ad8fc5457" committed_config { opid_index: -1 peers { permanent_uuid: "356eb15ea83947ae821c9f0ad8fc5457" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 43339 } } }
I20251024 20:02:00.316224 22088 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20251024 20:02:00.316203 22091 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "356eb15ea83947ae821c9f0ad8fc5457" committed_config { opid_index: -1 peers { permanent_uuid: "356eb15ea83947ae821c9f0ad8fc5457" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 43339 } } }
I20251024 20:02:00.316268 22091 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20251024 20:02:00.319346 22086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:43339 every 8 connection(s)
I20251024 20:02:00.319597 22093 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251024 20:02:00.319649 22093 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251024 20:02:00.319778 22089 heartbeater.cc:344] Connected to a master server at 127.18.153.126:36441
I20251024 20:02:00.319859 22089 heartbeater.cc:461] Registering TS with master...
I20251024 20:02:00.320019 22089 heartbeater.cc:507] Master 127.18.153.126:36441 requested a full tablet report, sending...
I20251024 20:02:00.320267 21847 ts_manager.cc:194] Re-registered known tserver with Master: 356eb15ea83947ae821c9f0ad8fc5457 (127.18.153.65:43339)
W20251024 20:02:00.320329 21847 catalog_manager.cc:5482] Ignoring report from unknown tablet 11111111111111111111111111111111
I20251024 20:02:00.320650 21847 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59286
I20251024 20:02:00.321033 22089 heartbeater.cc:499] Master 127.18.153.126:36441 was elected leader, sending a full tablet report...
W20251024 20:02:00.321255 21847 catalog_manager.cc:5482] Ignoring report from unknown tablet 11111111111111111111111111111111
I20251024 20:02:00.444274 19045 tablet_server.cc:178] TabletServer@127.18.153.65:43339 shutting down...
I20251024 20:02:00.446715 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:00.446898 19045 tablet_replica.cc:333] stopping tablet replica
I20251024 20:02:00.446972 19045 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 2 LEADER]: Raft consensus shutting down.
I20251024 20:02:00.447029 19045 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 356eb15ea83947ae821c9f0ad8fc5457 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:00.448702 19045 tablet_server.cc:195] TabletServer@127.18.153.65:43339 shutdown complete.
I20251024 20:02:00.449865 19045 master.cc:561] Master@127.18.153.126:36441 shutting down...
I20251024 20:02:00.451897 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:00.451963 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:00.452037 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P d8b0403f78c944c5825978d06404a74a: stopping tablet replica
I20251024 20:02:00.464053 19045 master.cc:583] Master@127.18.153.126:36441 shutdown complete.
[       OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (281 ms)
[ RUN      ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20251024 20:02:00.468047 19045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.153.126:40545
I20251024 20:02:00.468210 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:00.469285 22099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:00.469404 22100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:00.469372 22102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.469452 19045 server_base.cc:1047] running on GCE node
I20251024 20:02:00.469583 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:00.469616 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:00.469645 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336120469644 us; error 0 us; skew 500 ppm
I20251024 20:02:00.470242 19045 webserver.cc:492] Webserver started at http://127.18.153.126:44757/ using document root <none> and password file <none>
I20251024 20:02:00.470331 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:00.470374 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:00.470422 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:00.470728 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/master-0-root/instance:
uuid: "d2e2c4f7552b4def89e0477474484e7b"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.471712 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:00.472159 22107 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.472272 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.472316 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/master-0-root
uuid: "d2e2c4f7552b4def89e0477474484e7b"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.472359 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:00.480206 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:00.480404 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:00.482874 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.126:40545
I20251024 20:02:00.484556 22159 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.126:40545 every 8 connection(s)
I20251024 20:02:00.484679 22160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:00.485720 22160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b: Bootstrap starting.
I20251024 20:02:00.486001 22160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:00.486485 22160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b: No bootstrap required, opened a new log
I20251024 20:02:00.486604 22160 raft_consensus.cc:359] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e2c4f7552b4def89e0477474484e7b" member_type: VOTER }
I20251024 20:02:00.486656 22160 raft_consensus.cc:385] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:00.486671 22160 raft_consensus.cc:740] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e2c4f7552b4def89e0477474484e7b, State: Initialized, Role: FOLLOWER
I20251024 20:02:00.486728 22160 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e2c4f7552b4def89e0477474484e7b" member_type: VOTER }
I20251024 20:02:00.486769 22160 raft_consensus.cc:399] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:02:00.486784 22160 raft_consensus.cc:493] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:02:00.486812 22160 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:00.487298 22160 raft_consensus.cc:515] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e2c4f7552b4def89e0477474484e7b" member_type: VOTER }
I20251024 20:02:00.487358 22160 leader_election.cc:304] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2e2c4f7552b4def89e0477474484e7b; no voters: 
I20251024 20:02:00.487516 22160 leader_election.cc:290] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:02:00.487562 22163 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:02:00.487665 22160 sys_catalog.cc:565] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:02:00.487689 22163 raft_consensus.cc:697] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [term 1 LEADER]: Becoming Leader. State: Replica: d2e2c4f7552b4def89e0477474484e7b, State: Running, Role: LEADER
I20251024 20:02:00.487740 22163 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e2c4f7552b4def89e0477474484e7b" member_type: VOTER }
I20251024 20:02:00.488000 22165 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2e2c4f7552b4def89e0477474484e7b. Latest consensus state: current_term: 1 leader_uuid: "d2e2c4f7552b4def89e0477474484e7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e2c4f7552b4def89e0477474484e7b" member_type: VOTER } }
I20251024 20:02:00.488014 22164 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2e2c4f7552b4def89e0477474484e7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e2c4f7552b4def89e0477474484e7b" member_type: VOTER } }
I20251024 20:02:00.488106 22164 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [sys.catalog]: This master's current role is: LEADER
I20251024 20:02:00.488214 22165 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [sys.catalog]: This master's current role is: LEADER
I20251024 20:02:00.488869 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
W20251024 20:02:00.488955 22179 catalog_manager.cc:1568] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251024 20:02:00.488996 22179 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251024 20:02:00.489053 22169 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:02:00.489177 22169 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:02:00.489703 22169 catalog_manager.cc:1357] Generated new cluster ID: 7c391ddfea4a432bac721f7719de5b9e
I20251024 20:02:00.489741 22169 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:02:00.510201 22169 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 20:02:00.510675 22169 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:02:00.518173 22169 catalog_manager.cc:6022] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b: Generated new TSK 0
I20251024 20:02:00.518285 22169 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:02:00.520627 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:00.521941 22182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:00.521975 22184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.522032 19045 server_base.cc:1047] running on GCE node
W20251024 20:02:00.521965 22181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.522279 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:00.522317 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:00.522330 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336120522331 us; error 0 us; skew 500 ppm
I20251024 20:02:00.522886 19045 webserver.cc:492] Webserver started at http://127.18.153.65:44985/ using document root <none> and password file <none>
I20251024 20:02:00.522958 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:00.522991 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:00.523027 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:00.523322 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root/instance:
uuid: "5ec78931c03942e2b9979b4e47cc33e7"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.524257 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:02:00.524693 22189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.524842 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.524890 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "5ec78931c03942e2b9979b4e47cc33e7"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.524927 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:00.532567 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:00.532749 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:00.533138 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:02:00.533172 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.533195 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:02:00.533214 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.536796 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:40153
I20251024 20:02:00.536827 22255 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:40153 every 8 connection(s)
I20251024 20:02:00.537196 22256 heartbeater.cc:344] Connected to a master server at 127.18.153.126:40545
I20251024 20:02:00.537212 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251024 20:02:00.537263 22256 heartbeater.cc:461] Registering TS with master...
I20251024 20:02:00.537431 22256 heartbeater.cc:507] Master 127.18.153.126:40545 requested a full tablet report, sending...
I20251024 20:02:00.537645 22124 ts_manager.cc:194] Registered new tserver with Master: 5ec78931c03942e2b9979b4e47cc33e7 (127.18.153.65:40153)
W20251024 20:02:00.538488 22265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.538497 22124 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34554
W20251024 20:02:00.538579 22262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.538789 19045 server_base.cc:1047] running on GCE node
W20251024 20:02:00.538798 22263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.539038 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:00.539078 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:00.539105 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336120539105 us; error 0 us; skew 500 ppm
I20251024 20:02:00.539738 19045 webserver.cc:492] Webserver started at http://127.18.153.66:32815/ using document root <none> and password file <none>
I20251024 20:02:00.539825 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:00.539870 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:00.539913 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:00.540234 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-1-root/instance:
uuid: "43794b7608f847589fbb4529258042e0"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.541386 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.001s
I20251024 20:02:00.541919 22271 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.542037 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.001s	sys 0.000s
I20251024 20:02:00.542083 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-1-root
uuid: "43794b7608f847589fbb4529258042e0"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.542127 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:00.570963 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:00.571218 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:00.571628 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:02:00.571666 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.571698 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:02:00.571725 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.576128 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.66:32963
I20251024 20:02:00.576166 22341 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.66:32963 every 8 connection(s)
I20251024 20:02:00.576527 22342 heartbeater.cc:344] Connected to a master server at 127.18.153.126:40545
I20251024 20:02:00.576581 22342 heartbeater.cc:461] Registering TS with master...
I20251024 20:02:00.576685 22342 heartbeater.cc:507] Master 127.18.153.126:40545 requested a full tablet report, sending...
I20251024 20:02:00.576907 22124 ts_manager.cc:194] Registered new tserver with Master: 43794b7608f847589fbb4529258042e0 (127.18.153.66:32963)
I20251024 20:02:00.577389 22124 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34572
I20251024 20:02:00.577447 19045 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.001134949s
I20251024 20:02:00.584040 22220 tablet_service.cc:1505] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20251024 20:02:00.584187 22220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:00.585309 22353 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: Bootstrap starting.
I20251024 20:02:00.585731 22353 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:00.586212 22353 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: No bootstrap required, opened a new log
I20251024 20:02:00.586251 22353 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:00.586386 22353 raft_consensus.cc:359] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } }
I20251024 20:02:00.586438 22353 raft_consensus.cc:385] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:00.586469 22353 raft_consensus.cc:740] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec78931c03942e2b9979b4e47cc33e7, State: Initialized, Role: FOLLOWER
I20251024 20:02:00.586516 22353 consensus_queue.cc:260] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } }
I20251024 20:02:00.586560 22353 raft_consensus.cc:399] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:02:00.586576 22353 raft_consensus.cc:493] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:02:00.586594 22353 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:00.587112 22353 raft_consensus.cc:515] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } }
I20251024 20:02:00.587167 22353 leader_election.cc:304] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ec78931c03942e2b9979b4e47cc33e7; no voters: 
I20251024 20:02:00.587257 22353 leader_election.cc:290] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:02:00.587316 22355 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:02:00.587422 22256 heartbeater.cc:499] Master 127.18.153.126:40545 was elected leader, sending a full tablet report...
I20251024 20:02:00.587472 22355 raft_consensus.cc:697] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 1 LEADER]: Becoming Leader. State: Replica: 5ec78931c03942e2b9979b4e47cc33e7, State: Running, Role: LEADER
I20251024 20:02:00.587356 22353 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:00.587523 22355 consensus_queue.cc:237] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } }
I20251024 20:02:00.587704 22355 tablet_replica.cc:442] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: TxnStatusTablet state changed. Reason: New leader 5ec78931c03942e2b9979b4e47cc33e7. Latest consensus state: current_term: 1 leader_uuid: "5ec78931c03942e2b9979b4e47cc33e7" committed_config { opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } } }
I20251024 20:02:00.587759 22355 tablet_replica.cc:445] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: This TxnStatusTablet replica's current role is: LEADER
I20251024 20:02:00.587884 22359 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251024 20:02:00.587976 22359 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20251024 20:02:00.587991 22124 catalog_manager.cc:5482] Ignoring report from unknown tablet 11111111111111111111111111111111
I20251024 20:02:00.587950 22356 tablet_replica.cc:442] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ec78931c03942e2b9979b4e47cc33e7" committed_config { opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } } }
I20251024 20:02:00.588119 22356 tablet_replica.cc:445] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: This TxnStatusTablet replica's current role is: LEADER
W20251024 20:02:00.588435 22124 catalog_manager.cc:5482] Ignoring report from unknown tablet 11111111111111111111111111111111
I20251024 20:02:00.594817 22364 ts_tablet_manager.cc:933] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0: Initiating tablet copy from peer 5ec78931c03942e2b9979b4e47cc33e7 (127.18.153.65:40153)
I20251024 20:02:00.594980 22364 tablet_copy_client.cc:323] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0: tablet copy: Beginning tablet copy session from remote peer at address 127.18.153.65:40153
I20251024 20:02:00.597150 22240 tablet_copy_service.cc:140] P 5ec78931c03942e2b9979b4e47cc33e7: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer 43794b7608f847589fbb4529258042e0 ({username='slave'} at 127.0.0.1:39790)
I20251024 20:02:00.597208 22240 tablet_copy_service.cc:161] P 5ec78931c03942e2b9979b4e47cc33e7: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer 43794b7608f847589fbb4529258042e0 at {username='slave'} at 127.0.0.1:39790: session id = 43794b7608f847589fbb4529258042e0-11111111111111111111111111111111
I20251024 20:02:00.597652 22240 tablet_copy_source_session.cc:215] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: Tablet Copy: opened 0 blocks and 1 log segments
I20251024 20:02:00.597975 22364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:00.598974 22364 tablet_copy_client.cc:806] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0: tablet copy: Starting download of 0 data blocks...
I20251024 20:02:00.599107 22364 tablet_copy_client.cc:670] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0: tablet copy: Starting download of 1 WAL segments...
I20251024 20:02:00.599643 22364 tablet_copy_client.cc:538] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251024 20:02:00.600216 22364 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0: Bootstrap starting.
I20251024 20:02:00.601635 22364 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251024 20:02:00.601805 22364 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0: Bootstrap complete.
I20251024 20:02:00.601898 22364 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I20251024 20:02:00.602051 22364 raft_consensus.cc:359] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } }
I20251024 20:02:00.602113 22364 raft_consensus.cc:740] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 43794b7608f847589fbb4529258042e0, State: Initialized, Role: NON_PARTICIPANT
I20251024 20:02:00.602146 22364 consensus_queue.cc:260] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } }
I20251024 20:02:00.602300 22364 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0: Time spent starting tablet: real 0.000s	user 0.001s	sys 0.000s
I20251024 20:02:00.602344 22342 heartbeater.cc:499] Master 127.18.153.126:40545 was elected leader, sending a full tablet report...
I20251024 20:02:00.602339 22367 tablet_replica.cc:442] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } } }
I20251024 20:02:00.602470 22367 tablet_replica.cc:445] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20251024 20:02:00.602478 22240 tablet_copy_service.cc:342] P 5ec78931c03942e2b9979b4e47cc33e7: Request end of tablet copy session 43794b7608f847589fbb4529258042e0-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:39790
I20251024 20:02:00.602537 22240 tablet_copy_service.cc:434] P 5ec78931c03942e2b9979b4e47cc33e7: ending tablet copy session 43794b7608f847589fbb4529258042e0-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer 43794b7608f847589fbb4529258042e0
W20251024 20:02:00.602663 22124 catalog_manager.cc:5482] Ignoring report from unknown tablet 11111111111111111111111111111111
I20251024 20:02:00.606443 19045 tablet_server.cc:178] TabletServer@127.18.153.65:0 shutting down...
I20251024 20:02:00.608801 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:00.609028 19045 tablet_replica.cc:333] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: stopping tablet replica
I20251024 20:02:00.609097 19045 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:00.609143 19045 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:00.621300 19045 tablet_server.cc:195] TabletServer@127.18.153.65:0 shutdown complete.
I20251024 20:02:00.622624 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:00.623829 22371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:00.623891 22369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:00.623858 22368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.624056 19045 server_base.cc:1047] running on GCE node
I20251024 20:02:00.624121 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:00.624137 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:00.624150 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336120624150 us; error 0 us; skew 500 ppm
I20251024 20:02:00.624747 19045 webserver.cc:492] Webserver started at http://127.18.153.65:44985/ using document root <none> and password file <none>
I20251024 20:02:00.624814 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:00.624841 19045 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251024 20:02:00.625428 19045 fs_manager.cc:714] Time spent opening directory manager: real 0.000s	user 0.001s	sys 0.000s
I20251024 20:02:00.625895 22376 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.626029 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.626071 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "5ec78931c03942e2b9979b4e47cc33e7"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.626130 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:00.651984 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:00.652208 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:00.653295 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251024 20:02:00.653352 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
I20251024 20:02:00.653764 19045 ts_tablet_manager.cc:616] Registered 1 tablets
I20251024 20:02:00.653800 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.653815 22384 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: Bootstrap starting.
I20251024 20:02:00.655912 22384 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251024 20:02:00.656198 22384 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: Bootstrap complete.
I20251024 20:02:00.656316 22384 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
I20251024 20:02:00.656482 22384 raft_consensus.cc:359] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } }
I20251024 20:02:00.656561 22384 raft_consensus.cc:740] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec78931c03942e2b9979b4e47cc33e7, State: Initialized, Role: FOLLOWER
I20251024 20:02:00.656613 22384 consensus_queue.cc:260] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } }
I20251024 20:02:00.656651 22384 raft_consensus.cc:399] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:02:00.656675 22384 raft_consensus.cc:493] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:02:00.656698 22384 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:02:00.657331 22384 raft_consensus.cc:515] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } }
I20251024 20:02:00.657392 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:40153
I20251024 20:02:00.657440 22447 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:40153 every 8 connection(s)
I20251024 20:02:00.657397 22384 leader_election.cc:304] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ec78931c03942e2b9979b4e47cc33e7; no voters: 
I20251024 20:02:00.657613 22384 leader_election.cc:290] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [CANDIDATE]: Term 2 election: Requested vote from peers 
I20251024 20:02:00.657742 22384 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20251024 20:02:00.657799 22450 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 2 FOLLOWER]: Leader election won for term 2
I20251024 20:02:00.657842 22448 heartbeater.cc:344] Connected to a master server at 127.18.153.126:40545
I20251024 20:02:00.657963 22450 raft_consensus.cc:697] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 2 LEADER]: Becoming Leader. State: Replica: 5ec78931c03942e2b9979b4e47cc33e7, State: Running, Role: LEADER
I20251024 20:02:00.658022 22450 consensus_queue.cc:237] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } }
I20251024 20:02:00.658241 22452 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5ec78931c03942e2b9979b4e47cc33e7" committed_config { opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } } }
I20251024 20:02:00.658303 22452 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20251024 20:02:00.658269 22451 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 5ec78931c03942e2b9979b4e47cc33e7. Latest consensus state: current_term: 2 leader_uuid: "5ec78931c03942e2b9979b4e47cc33e7" committed_config { opid_index: -1 peers { permanent_uuid: "5ec78931c03942e2b9979b4e47cc33e7" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 40153 } } }
I20251024 20:02:00.658347 22451 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20251024 20:02:00.657953 22448 heartbeater.cc:461] Registering TS with master...
I20251024 20:02:00.658517 22456 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251024 20:02:00.658576 22456 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251024 20:02:00.658646 22448 heartbeater.cc:507] Master 127.18.153.126:40545 requested a full tablet report, sending...
I20251024 20:02:00.658893 22124 ts_manager.cc:194] Re-registered known tserver with Master: 5ec78931c03942e2b9979b4e47cc33e7 (127.18.153.65:40153)
W20251024 20:02:00.658946 22124 catalog_manager.cc:5482] Ignoring report from unknown tablet 11111111111111111111111111111111
I20251024 20:02:00.659420 22124 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34594
I20251024 20:02:00.659763 22448 heartbeater.cc:499] Master 127.18.153.126:40545 was elected leader, sending a full tablet report...
W20251024 20:02:00.660001 22124 catalog_manager.cc:5482] Ignoring report from unknown tablet 11111111111111111111111111111111
I20251024 20:02:00.786279 19045 tablet_server.cc:178] TabletServer@127.18.153.65:40153 shutting down...
I20251024 20:02:00.788802 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:00.788995 19045 tablet_replica.cc:333] stopping tablet replica
I20251024 20:02:00.789060 19045 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 2 LEADER]: Raft consensus shutting down.
I20251024 20:02:00.789126 19045 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 5ec78931c03942e2b9979b4e47cc33e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:00.790817 19045 tablet_server.cc:195] TabletServer@127.18.153.65:40153 shutdown complete.
I20251024 20:02:00.792025 19045 tablet_server.cc:178] TabletServer@127.18.153.66:0 shutting down...
I20251024 20:02:00.794430 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:00.794570 19045 tablet_replica.cc:333] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0: stopping tablet replica
I20251024 20:02:00.794615 19045 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20251024 20:02:00.794672 19045 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 43794b7608f847589fbb4529258042e0 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20251024 20:02:00.806499 19045 tablet_server.cc:195] TabletServer@127.18.153.66:0 shutdown complete.
I20251024 20:02:00.807713 19045 master.cc:561] Master@127.18.153.126:40545 shutting down...
I20251024 20:02:00.811522 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:00.811604 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:00.811659 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P d2e2c4f7552b4def89e0477474484e7b: stopping tablet replica
I20251024 20:02:00.823863 19045 master.cc:583] Master@127.18.153.126:40545 shutdown complete.
[       OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (360 ms)
[ RUN      ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20251024 20:02:00.828271 19045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.153.126:40059
I20251024 20:02:00.828442 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:00.829586 22462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:00.829655 22461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.829723 19045 server_base.cc:1047] running on GCE node
W20251024 20:02:00.829591 22464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.829905 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:00.829944 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:00.829957 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336120829957 us; error 0 us; skew 500 ppm
I20251024 20:02:00.830498 19045 webserver.cc:492] Webserver started at http://127.18.153.126:34685/ using document root <none> and password file <none>
I20251024 20:02:00.830580 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:00.830623 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:00.830680 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:00.830947 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1761336053687453-19045-0/minicluster-data/master-0-root/instance:
uuid: "6e38fdaf22184c9a9543dbeaa200c83b"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.831893 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:00.832373 22469 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.832469 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.832511 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1761336053687453-19045-0/minicluster-data/master-0-root
uuid: "6e38fdaf22184c9a9543dbeaa200c83b"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.832553 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1761336053687453-19045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1761336053687453-19045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1761336053687453-19045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:00.837276 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:00.837457 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:00.840035 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.126:40059
I20251024 20:02:00.841449 22521 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.126:40059 every 8 connection(s)
I20251024 20:02:00.841550 22522 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:00.842499 22522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b: Bootstrap starting.
I20251024 20:02:00.842746 22522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:00.843175 22522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b: No bootstrap required, opened a new log
I20251024 20:02:00.843272 22522 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e38fdaf22184c9a9543dbeaa200c83b" member_type: VOTER }
I20251024 20:02:00.843317 22522 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:00.843331 22522 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e38fdaf22184c9a9543dbeaa200c83b, State: Initialized, Role: FOLLOWER
I20251024 20:02:00.843454 22522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e38fdaf22184c9a9543dbeaa200c83b" member_type: VOTER }
I20251024 20:02:00.843493 22522 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:02:00.843508 22522 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:02:00.843523 22522 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:00.843971 22522 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e38fdaf22184c9a9543dbeaa200c83b" member_type: VOTER }
I20251024 20:02:00.844022 22522 leader_election.cc:304] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e38fdaf22184c9a9543dbeaa200c83b; no voters: 
I20251024 20:02:00.844107 22522 leader_election.cc:290] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:02:00.844151 22525 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:02:00.844255 22522 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:02:00.844269 22525 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [term 1 LEADER]: Becoming Leader. State: Replica: 6e38fdaf22184c9a9543dbeaa200c83b, State: Running, Role: LEADER
I20251024 20:02:00.844316 22525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e38fdaf22184c9a9543dbeaa200c83b" member_type: VOTER }
I20251024 20:02:00.844517 22526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e38fdaf22184c9a9543dbeaa200c83b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e38fdaf22184c9a9543dbeaa200c83b" member_type: VOTER } }
I20251024 20:02:00.844539 22527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e38fdaf22184c9a9543dbeaa200c83b. Latest consensus state: current_term: 1 leader_uuid: "6e38fdaf22184c9a9543dbeaa200c83b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e38fdaf22184c9a9543dbeaa200c83b" member_type: VOTER } }
I20251024 20:02:00.844645 22526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [sys.catalog]: This master's current role is: LEADER
I20251024 20:02:00.844674 22527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [sys.catalog]: This master's current role is: LEADER
I20251024 20:02:00.844847 22531 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:02:00.844982 22531 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:02:00.845516 22531 catalog_manager.cc:1357] Generated new cluster ID: 0bdbc3b482694c19bd057e631607fd33
I20251024 20:02:00.845552 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251024 20:02:00.845564 22531 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:02:00.864508 22531 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 20:02:00.864909 22531 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:02:00.867523 22531 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b: Generated new TSK 0
I20251024 20:02:00.867601 22531 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:02:00.877477 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:00.878777 22544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.878870 19045 server_base.cc:1047] running on GCE node
W20251024 20:02:00.878772 22546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:00.878939 22543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.879112 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:00.879146 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:00.879169 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336120879169 us; error 0 us; skew 500 ppm
I20251024 20:02:00.879779 19045 webserver.cc:492] Webserver started at http://127.18.153.65:35481/ using document root <none> and password file <none>
I20251024 20:02:00.879866 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:00.879920 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:00.879971 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:00.880251 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1761336053687453-19045-0/minicluster-data/ts-0-root/instance:
uuid: "853cb31b045f4cb9a88b6b631a041b7c"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.881129 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:02:00.881558 22551 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.881673 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.881713 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "853cb31b045f4cb9a88b6b631a041b7c"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.881762 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:00.896044 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:00.896266 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:00.896656 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:02:00.896693 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.896729 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:02:00.896814 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.900293 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:41229
I20251024 20:02:00.900319 22617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:41229 every 8 connection(s)
I20251024 20:02:00.900746 22618 heartbeater.cc:344] Connected to a master server at 127.18.153.126:40059
I20251024 20:02:00.900810 22618 heartbeater.cc:461] Registering TS with master...
I20251024 20:02:00.900936 22618 heartbeater.cc:507] Master 127.18.153.126:40059 requested a full tablet report, sending...
I20251024 20:02:00.901197 22486 ts_manager.cc:194] Registered new tserver with Master: 853cb31b045f4cb9a88b6b631a041b7c (127.18.153.65:41229)
I20251024 20:02:00.901611 19045 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.001124834s
I20251024 20:02:00.901923 22486 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49284
I20251024 20:02:00.908797 22582 tablet_service.cc:1505] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20251024 20:02:00.908932 22582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:00.910023 22634 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c: Bootstrap starting.
I20251024 20:02:00.910467 22634 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:00.910985 22634 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c: No bootstrap required, opened a new log
I20251024 20:02:00.911024 22634 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:00.911172 22634 raft_consensus.cc:359] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "853cb31b045f4cb9a88b6b631a041b7c" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 41229 } }
I20251024 20:02:00.911227 22634 raft_consensus.cc:385] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:00.911242 22634 raft_consensus.cc:740] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853cb31b045f4cb9a88b6b631a041b7c, State: Initialized, Role: FOLLOWER
I20251024 20:02:00.911288 22634 consensus_queue.cc:260] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "853cb31b045f4cb9a88b6b631a041b7c" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 41229 } }
I20251024 20:02:00.911329 22634 raft_consensus.cc:399] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:02:00.911358 22634 raft_consensus.cc:493] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:02:00.911409 22634 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:00.911942 22634 raft_consensus.cc:515] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "853cb31b045f4cb9a88b6b631a041b7c" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 41229 } }
I20251024 20:02:00.911999 22634 leader_election.cc:304] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 853cb31b045f4cb9a88b6b631a041b7c; no voters: 
I20251024 20:02:00.912099 22634 leader_election.cc:290] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:02:00.912171 22636 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:02:00.912252 22634 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:02:00.912308 22618 heartbeater.cc:499] Master 127.18.153.126:40059 was elected leader, sending a full tablet report...
I20251024 20:02:00.912432 22636 raft_consensus.cc:697] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [term 1 LEADER]: Becoming Leader. State: Replica: 853cb31b045f4cb9a88b6b631a041b7c, State: Running, Role: LEADER
I20251024 20:02:00.912475 22636 consensus_queue.cc:237] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "853cb31b045f4cb9a88b6b631a041b7c" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 41229 } }
I20251024 20:02:00.912712 22637 tablet_replica.cc:442] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "853cb31b045f4cb9a88b6b631a041b7c" committed_config { opid_index: -1 peers { permanent_uuid: "853cb31b045f4cb9a88b6b631a041b7c" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 41229 } } }
I20251024 20:02:00.912760 22638 tablet_replica.cc:442] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c: TxnStatusTablet state changed. Reason: New leader 853cb31b045f4cb9a88b6b631a041b7c. Latest consensus state: current_term: 1 leader_uuid: "853cb31b045f4cb9a88b6b631a041b7c" committed_config { opid_index: -1 peers { permanent_uuid: "853cb31b045f4cb9a88b6b631a041b7c" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 41229 } } }
I20251024 20:02:00.912971 22638 tablet_replica.cc:445] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c: This TxnStatusTablet replica's current role is: LEADER
W20251024 20:02:00.912983 22486 catalog_manager.cc:5482] Ignoring report from unknown tablet 11111111111111111111111111111111
I20251024 20:02:00.912892 22637 tablet_replica.cc:445] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c: This TxnStatusTablet replica's current role is: LEADER
I20251024 20:02:00.913157 22641 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251024 20:02:00.913204 22641 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20251024 20:02:00.913434 22486 catalog_manager.cc:5482] Ignoring report from unknown tablet 11111111111111111111111111111111
W20251024 20:02:00.928851 22622 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20251024 20:02:00.928925 22622 txn_status_manager.cc:244] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20251024 20:02:00.928946 22622 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20251024 20:02:00.929400 22622 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20251024 20:02:00.934298 22622 txn_status_manager.cc:244] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20251024 20:02:00.934356 22622 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20251024 20:02:00.934921 22622 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20251024 20:02:00.937556 22582 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:42230
W20251024 20:02:00.937908 22582 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:42230
W20251024 20:02:00.938156 22582 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:42230
W20251024 20:02:00.938366 22582 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:42230
W20251024 20:02:00.938570 22582 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:42230
W20251024 20:02:00.938779 22582 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:42230
W20251024 20:02:00.941540 22582 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:42236
W20251024 20:02:00.941929 22582 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:42236
W20251024 20:02:00.942205 22582 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:42236
W20251024 20:02:00.942523 22582 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:42236
W20251024 20:02:00.942742 22582 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:42236
W20251024 20:02:00.942958 22582 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:42236
I20251024 20:02:00.943418 19045 tablet_server.cc:178] TabletServer@127.18.153.65:0 shutting down...
I20251024 20:02:00.945737 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:00.945880 19045 tablet_replica.cc:333] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c: stopping tablet replica
I20251024 20:02:00.945956 19045 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:00.946013 19045 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 853cb31b045f4cb9a88b6b631a041b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:00.958359 19045 tablet_server.cc:195] TabletServer@127.18.153.65:0 shutdown complete.
I20251024 20:02:00.959643 19045 master.cc:561] Master@127.18.153.126:40059 shutting down...
I20251024 20:02:00.962284 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:00.962358 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:00.962435 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P 6e38fdaf22184c9a9543dbeaa200c83b: stopping tablet replica
I20251024 20:02:00.964042 19045 master.cc:583] Master@127.18.153.126:40059 shutdown complete.
[       OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (139 ms)
[ RUN      ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20251024 20:02:00.967553 19045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.153.126:42091
I20251024 20:02:00.967780 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:00.968724 22650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:00.968870 22649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.968861 19045 server_base.cc:1047] running on GCE node
W20251024 20:02:00.968806 22652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:00.969046 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:00.969076 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:00.969091 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336120969092 us; error 0 us; skew 500 ppm
I20251024 20:02:00.969606 19045 webserver.cc:492] Webserver started at http://127.18.153.126:36329/ using document root <none> and password file <none>
I20251024 20:02:00.969669 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:00.969702 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:00.969736 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:00.969959 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1761336053687453-19045-0/minicluster-data/master-0-root/instance:
uuid: "894c621f44194c7bb82480fc0139e8df"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.970757 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:02:00.971133 22657 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.971251 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:00.971302 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1761336053687453-19045-0/minicluster-data/master-0-root
uuid: "894c621f44194c7bb82480fc0139e8df"
format_stamp: "Formatted at 2025-10-24 20:02:00 on dist-test-slave-375r"
I20251024 20:02:00.971342 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1761336053687453-19045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1761336053687453-19045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1761336053687453-19045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:00.981963 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:00.982151 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:00.984735 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.126:42091
I20251024 20:02:00.986053 22709 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.126:42091 every 8 connection(s)
I20251024 20:02:00.986204 22710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:00.987241 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df: Bootstrap starting.
I20251024 20:02:00.987612 22710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:00.988101 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df: No bootstrap required, opened a new log
I20251024 20:02:00.988227 22710 raft_consensus.cc:359] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c621f44194c7bb82480fc0139e8df" member_type: VOTER }
I20251024 20:02:00.988278 22710 raft_consensus.cc:385] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:00.988292 22710 raft_consensus.cc:740] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894c621f44194c7bb82480fc0139e8df, State: Initialized, Role: FOLLOWER
I20251024 20:02:00.988343 22710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c621f44194c7bb82480fc0139e8df" member_type: VOTER }
I20251024 20:02:00.988384 22710 raft_consensus.cc:399] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:02:00.988399 22710 raft_consensus.cc:493] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:02:00.988427 22710 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:00.988879 22710 raft_consensus.cc:515] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c621f44194c7bb82480fc0139e8df" member_type: VOTER }
I20251024 20:02:00.988934 22710 leader_election.cc:304] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 894c621f44194c7bb82480fc0139e8df; no voters: 
I20251024 20:02:00.989035 22710 leader_election.cc:290] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:02:00.989101 22713 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:02:00.989187 22710 sys_catalog.cc:565] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:02:00.989214 22713 raft_consensus.cc:697] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [term 1 LEADER]: Becoming Leader. State: Replica: 894c621f44194c7bb82480fc0139e8df, State: Running, Role: LEADER
I20251024 20:02:00.989313 22713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c621f44194c7bb82480fc0139e8df" member_type: VOTER }
I20251024 20:02:00.989513 22713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 894c621f44194c7bb82480fc0139e8df. Latest consensus state: current_term: 1 leader_uuid: "894c621f44194c7bb82480fc0139e8df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c621f44194c7bb82480fc0139e8df" member_type: VOTER } }
I20251024 20:02:00.989550 22715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "894c621f44194c7bb82480fc0139e8df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894c621f44194c7bb82480fc0139e8df" member_type: VOTER } }
I20251024 20:02:00.989619 22713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [sys.catalog]: This master's current role is: LEADER
I20251024 20:02:00.989677 22715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [sys.catalog]: This master's current role is: LEADER
I20251024 20:02:00.989946 22719 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:02:00.990072 22719 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:02:00.990571 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251024 20:02:00.990640 22719 catalog_manager.cc:1357] Generated new cluster ID: f32a9006d8514773bf4cee8447f663b0
I20251024 20:02:00.990664 22719 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:02:01.007781 22719 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 20:02:01.008226 22719 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:02:01.011889 22719 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df: Generated new TSK 0
I20251024 20:02:01.011981 22719 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:02:01.022898 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:01.024200 22732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:01.024314 22734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:01.024371 19045 server_base.cc:1047] running on GCE node
W20251024 20:02:01.024308 22731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:01.024602 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:01.024639 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:01.024662 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336121024662 us; error 0 us; skew 500 ppm
I20251024 20:02:01.025285 19045 webserver.cc:492] Webserver started at http://127.18.153.65:36155/ using document root <none> and password file <none>
I20251024 20:02:01.025378 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:01.025424 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:01.025472 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:01.025758 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1761336053687453-19045-0/minicluster-data/ts-0-root/instance:
uuid: "82623497e712481f957bf9c977bfd09d"
format_stamp: "Formatted at 2025-10-24 20:02:01 on dist-test-slave-375r"
I20251024 20:02:01.026669 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:02:01.027087 22739 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.027205 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.027244 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "82623497e712481f957bf9c977bfd09d"
format_stamp: "Formatted at 2025-10-24 20:02:01 on dist-test-slave-375r"
I20251024 20:02:01.027287 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:01.032760 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:01.032960 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:01.033290 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:02:01.033340 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.033382 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:02:01.033413 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.037180 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:42201
I20251024 20:02:01.037209 22805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:42201 every 8 connection(s)
I20251024 20:02:01.037639 22806 heartbeater.cc:344] Connected to a master server at 127.18.153.126:42091
I20251024 20:02:01.037699 22806 heartbeater.cc:461] Registering TS with master...
I20251024 20:02:01.037822 22806 heartbeater.cc:507] Master 127.18.153.126:42091 requested a full tablet report, sending...
I20251024 20:02:01.038102 22674 ts_manager.cc:194] Registered new tserver with Master: 82623497e712481f957bf9c977bfd09d (127.18.153.65:42201)
I20251024 20:02:01.038509 19045 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.001123434s
I20251024 20:02:01.038774 22674 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54382
I20251024 20:02:01.046509 22770 tablet_service.cc:1505] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20251024 20:02:01.046654 22770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:01.047796 22822 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d: Bootstrap starting.
I20251024 20:02:01.048239 22822 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:01.048759 22822 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d: No bootstrap required, opened a new log
I20251024 20:02:01.048799 22822 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:01.048923 22822 raft_consensus.cc:359] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "82623497e712481f957bf9c977bfd09d" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42201 } }
I20251024 20:02:01.048971 22822 raft_consensus.cc:385] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:01.048990 22822 raft_consensus.cc:740] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82623497e712481f957bf9c977bfd09d, State: Initialized, Role: FOLLOWER
I20251024 20:02:01.049044 22822 consensus_queue.cc:260] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "82623497e712481f957bf9c977bfd09d" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42201 } }
I20251024 20:02:01.049090 22822 raft_consensus.cc:399] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:02:01.049105 22822 raft_consensus.cc:493] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:02:01.049124 22822 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:01.049634 22822 raft_consensus.cc:515] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "82623497e712481f957bf9c977bfd09d" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42201 } }
I20251024 20:02:01.049685 22822 leader_election.cc:304] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82623497e712481f957bf9c977bfd09d; no voters: 
I20251024 20:02:01.049773 22822 leader_election.cc:290] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:02:01.049829 22824 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:02:01.049890 22822 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:01.049958 22824 raft_consensus.cc:697] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [term 1 LEADER]: Becoming Leader. State: Replica: 82623497e712481f957bf9c977bfd09d, State: Running, Role: LEADER
I20251024 20:02:01.049973 22806 heartbeater.cc:499] Master 127.18.153.126:42091 was elected leader, sending a full tablet report...
I20251024 20:02:01.050012 22824 consensus_queue.cc:237] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "82623497e712481f957bf9c977bfd09d" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42201 } }
I20251024 20:02:01.050237 22825 tablet_replica.cc:442] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82623497e712481f957bf9c977bfd09d" committed_config { opid_index: -1 peers { permanent_uuid: "82623497e712481f957bf9c977bfd09d" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42201 } } }
I20251024 20:02:01.050339 22825 tablet_replica.cc:445] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d: This TxnStatusTablet replica's current role is: LEADER
W20251024 20:02:01.050465 22674 catalog_manager.cc:5482] Ignoring report from unknown tablet 11111111111111111111111111111111
I20251024 20:02:01.050276 22826 tablet_replica.cc:442] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d: TxnStatusTablet state changed. Reason: New leader 82623497e712481f957bf9c977bfd09d. Latest consensus state: current_term: 1 leader_uuid: "82623497e712481f957bf9c977bfd09d" committed_config { opid_index: -1 peers { permanent_uuid: "82623497e712481f957bf9c977bfd09d" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42201 } } }
I20251024 20:02:01.050560 22826 tablet_replica.cc:445] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d: This TxnStatusTablet replica's current role is: LEADER
I20251024 20:02:01.050558 22828 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251024 20:02:01.050648 22828 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20251024 20:02:01.050858 22674 catalog_manager.cc:5482] Ignoring report from unknown tablet 11111111111111111111111111111111
I20251024 20:02:01.054157 22770 tablet_service.cc:1505] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20251024 20:02:01.054270 22770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:01.055114 22822 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d: Bootstrap starting.
I20251024 20:02:01.055537 22822 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:01.056066 22822 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d: No bootstrap required, opened a new log
I20251024 20:02:01.056123 22822 ts_tablet_manager.cc:1403] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d: Time spent bootstrapping tablet: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:02:01.056255 22822 raft_consensus.cc:359] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "82623497e712481f957bf9c977bfd09d" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42201 } }
I20251024 20:02:01.056298 22822 raft_consensus.cc:385] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:01.056349 22822 raft_consensus.cc:740] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82623497e712481f957bf9c977bfd09d, State: Initialized, Role: FOLLOWER
I20251024 20:02:01.056399 22822 consensus_queue.cc:260] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "82623497e712481f957bf9c977bfd09d" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42201 } }
I20251024 20:02:01.056435 22822 raft_consensus.cc:399] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:02:01.056458 22822 raft_consensus.cc:493] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:02:01.056486 22822 raft_consensus.cc:3060] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:01.056967 22822 raft_consensus.cc:515] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "82623497e712481f957bf9c977bfd09d" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42201 } }
I20251024 20:02:01.057021 22822 leader_election.cc:304] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82623497e712481f957bf9c977bfd09d; no voters: 
I20251024 20:02:01.057070 22822 leader_election.cc:290] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:02:01.057096 22826 raft_consensus.cc:2804] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:02:01.057159 22826 raft_consensus.cc:697] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [term 1 LEADER]: Becoming Leader. State: Replica: 82623497e712481f957bf9c977bfd09d, State: Running, Role: LEADER
I20251024 20:02:01.057163 22822 ts_tablet_manager.cc:1434] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:02:01.057206 22826 consensus_queue.cc:237] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "82623497e712481f957bf9c977bfd09d" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 42201 } }
W20251024 20:02:01.057683 22674 catalog_manager.cc:5482] Ignoring report from unknown tablet 22222222222222222222222222222222
W20251024 20:02:01.058007 22674 catalog_manager.cc:5482] Ignoring report from unknown tablet 22222222222222222222222222222222
I20251024 20:02:01.060130 19045 tablet_server.cc:178] TabletServer@127.18.153.65:0 shutting down...
I20251024 20:02:01.062549 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:01.062708 19045 tablet_replica.cc:333] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d: stopping tablet replica
I20251024 20:02:01.062763 19045 raft_consensus.cc:2243] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:01.062815 19045 raft_consensus.cc:2272] T 22222222222222222222222222222222 P 82623497e712481f957bf9c977bfd09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:01.063071 19045 tablet_replica.cc:333] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d: stopping tablet replica
I20251024 20:02:01.063109 19045 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:01.063131 19045 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 82623497e712481f957bf9c977bfd09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:01.075168 19045 tablet_server.cc:195] TabletServer@127.18.153.65:0 shutdown complete.
I20251024 20:02:01.076581 19045 master.cc:561] Master@127.18.153.126:42091 shutting down...
I20251024 20:02:01.078954 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:01.079026 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:01.079082 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P 894c621f44194c7bb82480fc0139e8df: stopping tablet replica
I20251024 20:02:01.090821 19045 master.cc:583] Master@127.18.153.126:42091 shutdown complete.
[       OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (127 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (908 ms total)

[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN      ] FailedTabletsAreReplacedITest.OneReplica/0
I20251024 20:02:01.094617 19045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.153.126:35797
I20251024 20:02:01.094846 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:01.095942 22839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:01.095949 22838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:01.096006 22841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:01.096150 19045 server_base.cc:1047] running on GCE node
I20251024 20:02:01.096293 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:01.096328 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:01.096340 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336121096341 us; error 0 us; skew 500 ppm
I20251024 20:02:01.096881 19045 webserver.cc:492] Webserver started at http://127.18.153.126:34413/ using document root <none> and password file <none>
I20251024 20:02:01.096948 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:01.096982 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:01.097018 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:01.097265 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/master-0-root/instance:
uuid: "f51b793fdeca4f98b2e5d95363a18523"
format_stamp: "Formatted at 2025-10-24 20:02:01 on dist-test-slave-375r"
I20251024 20:02:01.098067 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:01.098469 22846 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.098587 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.098632 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/master-0-root
uuid: "f51b793fdeca4f98b2e5d95363a18523"
format_stamp: "Formatted at 2025-10-24 20:02:01 on dist-test-slave-375r"
I20251024 20:02:01.098675 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:01.119273 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:01.119495 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:01.121897 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.126:35797
I20251024 20:02:01.123304 22898 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.126:35797 every 8 connection(s)
I20251024 20:02:01.123457 22899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:01.124450 22899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523: Bootstrap starting.
I20251024 20:02:01.124712 22899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:01.125200 22899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523: No bootstrap required, opened a new log
I20251024 20:02:01.125304 22899 raft_consensus.cc:359] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51b793fdeca4f98b2e5d95363a18523" member_type: VOTER }
I20251024 20:02:01.125360 22899 raft_consensus.cc:385] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:01.125375 22899 raft_consensus.cc:740] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f51b793fdeca4f98b2e5d95363a18523, State: Initialized, Role: FOLLOWER
I20251024 20:02:01.125413 22899 consensus_queue.cc:260] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51b793fdeca4f98b2e5d95363a18523" member_type: VOTER }
I20251024 20:02:01.125455 22899 raft_consensus.cc:399] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:02:01.125476 22899 raft_consensus.cc:493] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:02:01.125504 22899 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:01.125964 22899 raft_consensus.cc:515] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51b793fdeca4f98b2e5d95363a18523" member_type: VOTER }
I20251024 20:02:01.126042 22899 leader_election.cc:304] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f51b793fdeca4f98b2e5d95363a18523; no voters: 
I20251024 20:02:01.126168 22899 leader_election.cc:290] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:02:01.126223 22902 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:02:01.126320 22899 sys_catalog.cc:565] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:02:01.126377 22902 raft_consensus.cc:697] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [term 1 LEADER]: Becoming Leader. State: Replica: f51b793fdeca4f98b2e5d95363a18523, State: Running, Role: LEADER
I20251024 20:02:01.126426 22902 consensus_queue.cc:237] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51b793fdeca4f98b2e5d95363a18523" member_type: VOTER }
I20251024 20:02:01.126678 22904 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f51b793fdeca4f98b2e5d95363a18523. Latest consensus state: current_term: 1 leader_uuid: "f51b793fdeca4f98b2e5d95363a18523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51b793fdeca4f98b2e5d95363a18523" member_type: VOTER } }
I20251024 20:02:01.126657 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f51b793fdeca4f98b2e5d95363a18523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51b793fdeca4f98b2e5d95363a18523" member_type: VOTER } }
I20251024 20:02:01.126760 22904 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [sys.catalog]: This master's current role is: LEADER
I20251024 20:02:01.126784 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [sys.catalog]: This master's current role is: LEADER
I20251024 20:02:01.126942 22908 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:02:01.127108 22908 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:02:01.127651 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251024 20:02:01.127910 22908 catalog_manager.cc:1357] Generated new cluster ID: 13f438822d5142398634d582bdfd80a4
I20251024 20:02:01.127955 22908 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:02:01.143198 22908 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 20:02:01.143709 22908 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:02:01.148043 22908 catalog_manager.cc:6022] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523: Generated new TSK 0
I20251024 20:02:01.148144 22908 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:02:01.159602 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:01.160916 22923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:01.160987 22920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:01.161129 19045 server_base.cc:1047] running on GCE node
W20251024 20:02:01.160989 22921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:01.161298 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:01.161335 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:01.161348 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336121161349 us; error 0 us; skew 500 ppm
I20251024 20:02:01.161970 19045 webserver.cc:492] Webserver started at http://127.18.153.65:44285/ using document root <none> and password file <none>
I20251024 20:02:01.162057 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:01.162091 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:01.162128 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:01.162387 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-0-root/instance:
uuid: "5d7f551b0d494bf69275a3f42b1d7ba4"
format_stamp: "Formatted at 2025-10-24 20:02:01 on dist-test-slave-375r"
I20251024 20:02:01.163225 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:02:01.163736 22928 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.163864 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.163911 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "5d7f551b0d494bf69275a3f42b1d7ba4"
format_stamp: "Formatted at 2025-10-24 20:02:01 on dist-test-slave-375r"
I20251024 20:02:01.163960 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:01.168826 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:01.169015 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:01.169147 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:02:01.169340 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:02:01.169368 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.169399 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:02:01.169426 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.172327 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:39037
I20251024 20:02:01.172361 22991 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:39037 every 8 connection(s)
I20251024 20:02:01.172686 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:01.173828 22996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:01.173844 22997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:01.174038 22999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:01.174674 19045 server_base.cc:1047] running on GCE node
I20251024 20:02:01.174789 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:01.174818 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:01.174840 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336121174840 us; error 0 us; skew 500 ppm
I20251024 20:02:01.175529 19045 webserver.cc:492] Webserver started at http://127.18.153.66:45325/ using document root <none> and password file <none>
I20251024 20:02:01.175622 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:01.175673 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:01.175727 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:01.176012 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-1-root/instance:
uuid: "6b58eaa4961f4c0897f0fff958a81201"
format_stamp: "Formatted at 2025-10-24 20:02:01 on dist-test-slave-375r"
I20251024 20:02:01.176152 22992 heartbeater.cc:344] Connected to a master server at 127.18.153.126:35797
I20251024 20:02:01.176213 22992 heartbeater.cc:461] Registering TS with master...
I20251024 20:02:01.176306 22992 heartbeater.cc:507] Master 127.18.153.126:35797 requested a full tablet report, sending...
I20251024 20:02:01.176534 22863 ts_manager.cc:194] Registered new tserver with Master: 5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037)
I20251024 20:02:01.177063 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:02:01.177328 22863 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50932
I20251024 20:02:01.177573 23004 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.177678 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.177729 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-1-root
uuid: "6b58eaa4961f4c0897f0fff958a81201"
format_stamp: "Formatted at 2025-10-24 20:02:01 on dist-test-slave-375r"
I20251024 20:02:01.177769 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:01.188755 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:01.189008 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:01.189179 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:02:01.189373 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:02:01.189401 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.189437 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:02:01.189464 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.192763 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.66:36927
I20251024 20:02:01.192828 23067 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.66:36927 every 8 connection(s)
I20251024 20:02:01.193149 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:01.194365 23072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:01.194439 23074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:01.194396 23071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:01.194900 19045 server_base.cc:1047] running on GCE node
I20251024 20:02:01.195020 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:01.195055 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:01.195075 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336121195074 us; error 0 us; skew 500 ppm
I20251024 20:02:01.195817 19045 webserver.cc:492] Webserver started at http://127.18.153.67:37937/ using document root <none> and password file <none>
I20251024 20:02:01.195914 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:01.195963 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:01.196017 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:01.196374 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-2-root/instance:
uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf"
format_stamp: "Formatted at 2025-10-24 20:02:01 on dist-test-slave-375r"
I20251024 20:02:01.196394 23068 heartbeater.cc:344] Connected to a master server at 127.18.153.126:35797
I20251024 20:02:01.196470 23068 heartbeater.cc:461] Registering TS with master...
I20251024 20:02:01.196591 23068 heartbeater.cc:507] Master 127.18.153.126:35797 requested a full tablet report, sending...
I20251024 20:02:01.196818 22863 ts_manager.cc:194] Registered new tserver with Master: 6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66:36927)
I20251024 20:02:01.197319 22863 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50948
I20251024 20:02:01.197458 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:01.197933 23079 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.198046 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.198087 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-2-root
uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf"
format_stamp: "Formatted at 2025-10-24 20:02:01 on dist-test-slave-375r"
I20251024 20:02:01.198129 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:01.208943 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:01.209211 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:01.209393 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:02:01.209605 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:02:01.209635 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.209666 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:02:01.209692 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.212890 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.67:40675
I20251024 20:02:01.212950 23142 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.67:40675 every 8 connection(s)
I20251024 20:02:01.216082 23143 heartbeater.cc:344] Connected to a master server at 127.18.153.126:35797
I20251024 20:02:01.216150 23143 heartbeater.cc:461] Registering TS with master...
I20251024 20:02:01.216256 23143 heartbeater.cc:507] Master 127.18.153.126:35797 requested a full tablet report, sending...
I20251024 20:02:01.216495 22863 ts_manager.cc:194] Registered new tserver with Master: 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675)
I20251024 20:02:01.216822 22863 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50958
I20251024 20:02:01.217502 19045 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004427867s
I20251024 20:02:01.221800 19045 test_util.cc:276] Using random seed: 97903269
I20251024 20:02:01.225979 22863 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50976:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20251024 20:02:01.230746 23032 tablet_service.cc:1505] Processing CreateTablet for tablet f0b28aaca8d94b7abbe668af88781aaa (DEFAULT_TABLE table=test-workload [id=fb4aa255e8e44fe5b565f2b0d2b39caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 20:02:01.230926 23032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0b28aaca8d94b7abbe668af88781aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:01.231386 22956 tablet_service.cc:1505] Processing CreateTablet for tablet f0b28aaca8d94b7abbe668af88781aaa (DEFAULT_TABLE table=test-workload [id=fb4aa255e8e44fe5b565f2b0d2b39caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 20:02:01.231495 22956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0b28aaca8d94b7abbe668af88781aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:01.232781 23163 tablet_bootstrap.cc:492] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201: Bootstrap starting.
I20251024 20:02:01.232791 23164 tablet_bootstrap.cc:492] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4: Bootstrap starting.
I20251024 20:02:01.232921 23107 tablet_service.cc:1505] Processing CreateTablet for tablet f0b28aaca8d94b7abbe668af88781aaa (DEFAULT_TABLE table=test-workload [id=fb4aa255e8e44fe5b565f2b0d2b39caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 20:02:01.233084 23107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0b28aaca8d94b7abbe668af88781aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:01.233399 23163 tablet_bootstrap.cc:654] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:01.233399 23164 tablet_bootstrap.cc:654] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:01.234076 23164 tablet_bootstrap.cc:492] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4: No bootstrap required, opened a new log
I20251024 20:02:01.234123 23164 ts_tablet_manager.cc:1403] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:01.234150 23167 tablet_bootstrap.cc:492] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: Bootstrap starting.
I20251024 20:02:01.234155 23163 tablet_bootstrap.cc:492] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201: No bootstrap required, opened a new log
I20251024 20:02:01.234232 23163 ts_tablet_manager.cc:1403] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:01.234365 23163 raft_consensus.cc:359] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:01.234437 23163 raft_consensus.cc:385] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:01.234416 23164 raft_consensus.cc:359] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:01.234465 23164 raft_consensus.cc:385] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:01.234458 23163 raft_consensus.cc:740] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b58eaa4961f4c0897f0fff958a81201, State: Initialized, Role: FOLLOWER
I20251024 20:02:01.234551 23164 raft_consensus.cc:740] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d7f551b0d494bf69275a3f42b1d7ba4, State: Initialized, Role: FOLLOWER
I20251024 20:02:01.234622 23164 consensus_queue.cc:260] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:01.234637 23167 tablet_bootstrap.cc:654] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:01.234622 23163 consensus_queue.cc:260] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:01.234822 23164 ts_tablet_manager.cc:1434] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:01.234876 23068 heartbeater.cc:499] Master 127.18.153.126:35797 was elected leader, sending a full tablet report...
I20251024 20:02:01.234830 23163 ts_tablet_manager.cc:1434] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:01.234944 22992 heartbeater.cc:499] Master 127.18.153.126:35797 was elected leader, sending a full tablet report...
I20251024 20:02:01.235410 23167 tablet_bootstrap.cc:492] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: No bootstrap required, opened a new log
I20251024 20:02:01.235464 23167 ts_tablet_manager.cc:1403] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:01.235646 23167 raft_consensus.cc:359] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:01.235739 23167 raft_consensus.cc:385] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:01.235770 23167 raft_consensus.cc:740] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c499cd2fd34cb0aa6c2a8feadc73bf, State: Initialized, Role: FOLLOWER
I20251024 20:02:01.235850 23167 consensus_queue.cc:260] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:01.236014 23167 ts_tablet_manager.cc:1434] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: Time spent starting tablet: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:01.236047 23143 heartbeater.cc:499] Master 127.18.153.126:35797 was elected leader, sending a full tablet report...
I20251024 20:02:01.501106 23169 raft_consensus.cc:493] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 20:02:01.501220 23169 raft_consensus.cc:515] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:01.501374 23171 raft_consensus.cc:493] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 20:02:01.501479 23169 leader_election.cc:290] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675), 5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037)
I20251024 20:02:01.501463 23171 raft_consensus.cc:515] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:01.501796 23171 leader_election.cc:290] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037), 6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66:36927)
I20251024 20:02:01.505023 22966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0b28aaca8d94b7abbe668af88781aaa" candidate_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" is_pre_election: true
I20251024 20:02:01.505066 23117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0b28aaca8d94b7abbe668af88781aaa" candidate_uuid: "6b58eaa4961f4c0897f0fff958a81201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" is_pre_election: true
I20251024 20:02:01.505019 23042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0b28aaca8d94b7abbe668af88781aaa" candidate_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b58eaa4961f4c0897f0fff958a81201" is_pre_election: true
I20251024 20:02:01.505216 23042 raft_consensus.cc:2468] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c499cd2fd34cb0aa6c2a8feadc73bf in term 0.
I20251024 20:02:01.505280 22965 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0b28aaca8d94b7abbe668af88781aaa" candidate_uuid: "6b58eaa4961f4c0897f0fff958a81201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" is_pre_election: true
I20251024 20:02:01.505404 23080 leader_election.cc:304] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c499cd2fd34cb0aa6c2a8feadc73bf, 6b58eaa4961f4c0897f0fff958a81201; no voters: 
I20251024 20:02:01.505203 23117 raft_consensus.cc:2468] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b58eaa4961f4c0897f0fff958a81201 in term 0.
I20251024 20:02:01.505508 23171 raft_consensus.cc:2804] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 20:02:01.505561 23171 raft_consensus.cc:493] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 20:02:01.505189 22966 raft_consensus.cc:2468] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c499cd2fd34cb0aa6c2a8feadc73bf in term 0.
I20251024 20:02:01.505600 23171 raft_consensus.cc:3060] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:01.505621 23006 leader_election.cc:304] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 11c499cd2fd34cb0aa6c2a8feadc73bf, 6b58eaa4961f4c0897f0fff958a81201; no voters: 5d7f551b0d494bf69275a3f42b1d7ba4
I20251024 20:02:01.505748 23169 raft_consensus.cc:2804] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 20:02:01.505779 23169 raft_consensus.cc:493] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 20:02:01.505805 23169 raft_consensus.cc:3060] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:01.506332 23171 raft_consensus.cc:515] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:01.506453 23171 leader_election.cc:290] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [CANDIDATE]: Term 1 election: Requested vote from peers 5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037), 6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66:36927)
I20251024 20:02:01.506438 23169 raft_consensus.cc:515] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:01.506618 22966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0b28aaca8d94b7abbe668af88781aaa" candidate_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4"
I20251024 20:02:01.506641 23042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0b28aaca8d94b7abbe668af88781aaa" candidate_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b58eaa4961f4c0897f0fff958a81201"
I20251024 20:02:01.506683 22966 raft_consensus.cc:3060] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:01.506714 23042 raft_consensus.cc:2393] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 11c499cd2fd34cb0aa6c2a8feadc73bf in current term 1: Already voted for candidate 6b58eaa4961f4c0897f0fff958a81201 in this term.
I20251024 20:02:01.506650 23169 leader_election.cc:290] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [CANDIDATE]: Term 1 election: Requested vote from peers 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675), 5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037)
I20251024 20:02:01.506763 22965 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0b28aaca8d94b7abbe668af88781aaa" candidate_uuid: "6b58eaa4961f4c0897f0fff958a81201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4"
I20251024 20:02:01.506991 23117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0b28aaca8d94b7abbe668af88781aaa" candidate_uuid: "6b58eaa4961f4c0897f0fff958a81201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf"
I20251024 20:02:01.507042 23117 raft_consensus.cc:2393] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b58eaa4961f4c0897f0fff958a81201 in current term 1: Already voted for candidate 11c499cd2fd34cb0aa6c2a8feadc73bf in this term.
I20251024 20:02:01.507170 23006 leader_election.cc:304] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b58eaa4961f4c0897f0fff958a81201; no voters: 11c499cd2fd34cb0aa6c2a8feadc73bf, 5d7f551b0d494bf69275a3f42b1d7ba4
I20251024 20:02:01.507274 23169 raft_consensus.cc:2749] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251024 20:02:01.507409 22966 raft_consensus.cc:2468] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c499cd2fd34cb0aa6c2a8feadc73bf in term 1.
I20251024 20:02:01.507572 23082 leader_election.cc:304] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 11c499cd2fd34cb0aa6c2a8feadc73bf, 5d7f551b0d494bf69275a3f42b1d7ba4; no voters: 6b58eaa4961f4c0897f0fff958a81201
I20251024 20:02:01.507701 23171 raft_consensus.cc:2804] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:02:01.507845 23171 raft_consensus.cc:697] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 1 LEADER]: Becoming Leader. State: Replica: 11c499cd2fd34cb0aa6c2a8feadc73bf, State: Running, Role: LEADER
I20251024 20:02:01.507889 23171 consensus_queue.cc:237] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:01.508374 22863 catalog_manager.cc:5649] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67). New cstate: current_term: 1 leader_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } } }
I20251024 20:02:01.534806 23107 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20251024 20:02:01.534951 22956 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:01.535581 23032 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 f51b793fdeca4f98b2e5d95363a18523 | 127.18.153.126:35797 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 11c499cd2fd34cb0aa6c2a8feadc73bf | 127.18.153.67:40675 | HEALTHY | <none>   |       1        |       0
 5d7f551b0d494bf69275a3f42b1d7ba4 | 127.18.153.65:39037 | HEALTHY | <none>   |       0        |       0
 6b58eaa4961f4c0897f0fff958a81201 | 127.18.153.66:36927 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet f0b28aaca8d94b7abbe668af88781aaa of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
  11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): RUNNING [LEADER]
  5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037): RUNNING
  6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66:36927): RUNNING
All reported replicas are:
  A = 11c499cd2fd34cb0aa6c2a8feadc73bf
  B = 5d7f551b0d494bf69275a3f42b1d7ba4
  C = 6b58eaa4961f4c0897f0fff958a81201
The consensus matrix is:
 Config source |   Replicas   | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A*  B   C    |              |              | Yes
 A             | A*  B   C    | 1            | -1           | Yes
 B             | A   B   C    | 1            | -1           | Yes
 C             | A   B   C    | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |       Status       | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | CONSENSUS_MISMATCH | 1             | 0       | 0          | 0                | 1

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:01.544560 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:01.716617 22956 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:01.716687 23032 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:01.716889 23107 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 f51b793fdeca4f98b2e5d95363a18523 | 127.18.153.126:35797 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 11c499cd2fd34cb0aa6c2a8feadc73bf | 127.18.153.67:40675 | HEALTHY | <none>   |       1        |       0
 5d7f551b0d494bf69275a3f42b1d7ba4 | 127.18.153.65:39037 | HEALTHY | <none>   |       0        |       0
 6b58eaa4961f4c0897f0fff958a81201 | 127.18.153.66:36927 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet f0b28aaca8d94b7abbe668af88781aaa of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
  11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): RUNNING [LEADER]
  5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037): RUNNING
  6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66:36927): RUNNING
All reported replicas are:
  A = 11c499cd2fd34cb0aa6c2a8feadc73bf
  B = 5d7f551b0d494bf69275a3f42b1d7ba4
  C = 6b58eaa4961f4c0897f0fff958a81201
The consensus matrix is:
 Config source |   Replicas   | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A*  B   C    |              |              | Yes
 A             | A*  B   C    | 1            | -1           | Yes
 B             | A   B   C    | 1            | -1           | Yes
 C             | A   B   C    | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |       Status       | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | CONSENSUS_MISMATCH | 1             | 0       | 0          | 0                | 1

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:01.725479 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:01.974848 23107 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20251024 20:02:01.974910 22956 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:01.974936 23032 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 f51b793fdeca4f98b2e5d95363a18523 | 127.18.153.126:35797 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 11c499cd2fd34cb0aa6c2a8feadc73bf | 127.18.153.67:40675 | HEALTHY | <none>   |       1        |       0
 5d7f551b0d494bf69275a3f42b1d7ba4 | 127.18.153.65:39037 | HEALTHY | <none>   |       0        |       0
 6b58eaa4961f4c0897f0fff958a81201 | 127.18.153.66:36927 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet f0b28aaca8d94b7abbe668af88781aaa of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
  11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): RUNNING [LEADER]
  5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037): RUNNING
  6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66:36927): RUNNING
All reported replicas are:
  A = 11c499cd2fd34cb0aa6c2a8feadc73bf
  B = 5d7f551b0d494bf69275a3f42b1d7ba4
  C = 6b58eaa4961f4c0897f0fff958a81201
The consensus matrix is:
 Config source |   Replicas   | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A*  B   C    |              |              | Yes
 A             | A*  B   C    | 1            | -1           | Yes
 B             | A   B   C    | 1            | -1           | Yes
 C             | A   B   C    | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |       Status       | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | CONSENSUS_MISMATCH | 1             | 0       | 0          | 0                | 1

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:01.983599 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:02.010468 23249 consensus_queue.cc:1048] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:02:02.011976 23249 consensus_queue.cc:1048] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:02:02.345467 23107 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20251024 20:02:02.345527 23032 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:02.345793 22956 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 f51b793fdeca4f98b2e5d95363a18523 | 127.18.153.126:35797 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 11c499cd2fd34cb0aa6c2a8feadc73bf | 127.18.153.67:40675 | HEALTHY | <none>   |       1        |       0
 5d7f551b0d494bf69275a3f42b1d7ba4 | 127.18.153.65:39037 | HEALTHY | <none>   |       0        |       0
 6b58eaa4961f4c0897f0fff958a81201 | 127.18.153.66:36927 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
     Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20251024 20:02:02.354867 19045 log_verifier.cc:126] Checking tablet f0b28aaca8d94b7abbe668af88781aaa
I20251024 20:02:02.372979 19045 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f0b28aaca8d94b7abbe668af88781aaa
I20251024 20:02:02.373113 19045 tablet_replica.cc:333] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: stopping tablet replica
I20251024 20:02:02.373145 19045 raft_consensus.cc:2243] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:02.373212 19045 raft_consensus.cc:2272] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:02.397053 23032 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:02.397562 22956 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:02.397990 23107 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 f51b793fdeca4f98b2e5d95363a18523 | 127.18.153.126:35797 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 11c499cd2fd34cb0aa6c2a8feadc73bf | 127.18.153.67:40675 | HEALTHY | <none>   |       0        |       0
 5d7f551b0d494bf69275a3f42b1d7ba4 | 127.18.153.65:39037 | HEALTHY | <none>   |       0        |       0
 6b58eaa4961f4c0897f0fff958a81201 | 127.18.153.66:36927 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet f0b28aaca8d94b7abbe668af88781aaa of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
  5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037): RUNNING
  6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66:36927): RUNNING
All reported replicas are:
  A = 11c499cd2fd34cb0aa6c2a8feadc73bf
  B = 5d7f551b0d494bf69275a3f42b1d7ba4
  C = 6b58eaa4961f4c0897f0fff958a81201
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A*  B   C              |              |              | Yes
 A             | [config not available] |              |              | 
 B             | A*  B   C              | 1            | -1           | Yes
 C             | A*  B   C              | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:02.406024 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:02.578863 23107 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:02.579135 23032 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:02.579141 22956 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 f51b793fdeca4f98b2e5d95363a18523 | 127.18.153.126:35797 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 11c499cd2fd34cb0aa6c2a8feadc73bf | 127.18.153.67:40675 | HEALTHY | <none>   |       0        |       0
 5d7f551b0d494bf69275a3f42b1d7ba4 | 127.18.153.65:39037 | HEALTHY | <none>   |       0        |       0
 6b58eaa4961f4c0897f0fff958a81201 | 127.18.153.66:36927 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet f0b28aaca8d94b7abbe668af88781aaa of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
  5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037): RUNNING
  6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66:36927): RUNNING
All reported replicas are:
  A = 11c499cd2fd34cb0aa6c2a8feadc73bf
  B = 5d7f551b0d494bf69275a3f42b1d7ba4
  C = 6b58eaa4961f4c0897f0fff958a81201
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A*  B   C              |              |              | Yes
 A             | [config not available] |              |              | 
 B             | A*  B   C              | 1            | -1           | Yes
 C             | A*  B   C              | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:02.587204 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:02.834709 23107 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:02.834860 23032 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:02.835183 22956 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 f51b793fdeca4f98b2e5d95363a18523 | 127.18.153.126:35797 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 11c499cd2fd34cb0aa6c2a8feadc73bf | 127.18.153.67:40675 | HEALTHY | <none>   |       0        |       0
 5d7f551b0d494bf69275a3f42b1d7ba4 | 127.18.153.65:39037 | HEALTHY | <none>   |       0        |       0
 6b58eaa4961f4c0897f0fff958a81201 | 127.18.153.66:36927 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet f0b28aaca8d94b7abbe668af88781aaa of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
  5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037): RUNNING
  6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66:36927): RUNNING
All reported replicas are:
  A = 11c499cd2fd34cb0aa6c2a8feadc73bf
  B = 5d7f551b0d494bf69275a3f42b1d7ba4
  C = 6b58eaa4961f4c0897f0fff958a81201
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A*  B   C              |              |              | Yes
 A             | [config not available] |              |              | 
 B             | A*  B   C              | 1            | -1           | Yes
 C             | A*  B   C              | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:02.843743 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:03.209452 22956 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:03.210170 23032 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:03.210265 23107 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 f51b793fdeca4f98b2e5d95363a18523 | 127.18.153.126:35797 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 11c499cd2fd34cb0aa6c2a8feadc73bf | 127.18.153.67:40675 | HEALTHY | <none>   |       0        |       0
 5d7f551b0d494bf69275a3f42b1d7ba4 | 127.18.153.65:39037 | HEALTHY | <none>   |       0        |       0
 6b58eaa4961f4c0897f0fff958a81201 | 127.18.153.66:36927 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet f0b28aaca8d94b7abbe668af88781aaa of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
  5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037): RUNNING
  6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66:36927): RUNNING
All reported replicas are:
  A = 11c499cd2fd34cb0aa6c2a8feadc73bf
  B = 5d7f551b0d494bf69275a3f42b1d7ba4
  C = 6b58eaa4961f4c0897f0fff958a81201
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A*  B   C              |              |              | Yes
 A             | [config not available] |              |              | 
 B             | A*  B   C              | 1            | -1           | Yes
 C             | A*  B   C              | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:03.219089 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:03.513001 23373 raft_consensus.cc:493] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 11c499cd2fd34cb0aa6c2a8feadc73bf)
I20251024 20:02:03.513130 23373 raft_consensus.cc:515] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:03.513258 23374 raft_consensus.cc:493] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 11c499cd2fd34cb0aa6c2a8feadc73bf)
I20251024 20:02:03.513343 23373 leader_election.cc:290] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675), 5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037)
I20251024 20:02:03.513525 23117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0b28aaca8d94b7abbe668af88781aaa" candidate_uuid: "6b58eaa4961f4c0897f0fff958a81201" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" is_pre_election: true
I20251024 20:02:03.513334 23374 raft_consensus.cc:515] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:03.513536 22966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0b28aaca8d94b7abbe668af88781aaa" candidate_uuid: "6b58eaa4961f4c0897f0fff958a81201" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" is_pre_election: true
I20251024 20:02:03.513628 22966 raft_consensus.cc:2468] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b58eaa4961f4c0897f0fff958a81201 in term 1.
I20251024 20:02:03.513767 23007 leader_election.cc:304] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d7f551b0d494bf69275a3f42b1d7ba4, 6b58eaa4961f4c0897f0fff958a81201; no voters: 
W20251024 20:02:03.513818 23006 leader_election.cc:343] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): Illegal state: cannot vote while shut down
I20251024 20:02:03.513847 23373 raft_consensus.cc:2804] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251024 20:02:03.513898 23373 raft_consensus.cc:493] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 11c499cd2fd34cb0aa6c2a8feadc73bf)
I20251024 20:02:03.513988 23373 raft_consensus.cc:3060] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:02:03.514154 23374 leader_election.cc:290] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675), 6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66:36927)
I20251024 20:02:03.514950 23373 raft_consensus.cc:515] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:03.515210 22966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0b28aaca8d94b7abbe668af88781aaa" candidate_uuid: "6b58eaa4961f4c0897f0fff958a81201" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4"
I20251024 20:02:03.515210 23117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0b28aaca8d94b7abbe668af88781aaa" candidate_uuid: "6b58eaa4961f4c0897f0fff958a81201" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf"
I20251024 20:02:03.515285 22966 raft_consensus.cc:3060] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 1 FOLLOWER]: Advancing to term 2
W20251024 20:02:03.515419 23006 leader_election.cc:343] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): Illegal state: cannot vote while shut down
I20251024 20:02:03.515913 22966 raft_consensus.cc:2468] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b58eaa4961f4c0897f0fff958a81201 in term 2.
I20251024 20:02:03.516064 23007 leader_election.cc:304] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d7f551b0d494bf69275a3f42b1d7ba4, 6b58eaa4961f4c0897f0fff958a81201; no voters: 11c499cd2fd34cb0aa6c2a8feadc73bf
I20251024 20:02:03.516223 23373 leader_election.cc:290] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [CANDIDATE]: Term 2 election: Requested vote from peers 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675), 5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037)
I20251024 20:02:03.516254 23377 raft_consensus.cc:2804] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 2 FOLLOWER]: Leader election won for term 2
I20251024 20:02:03.516391 23377 raft_consensus.cc:697] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 2 LEADER]: Becoming Leader. State: Replica: 6b58eaa4961f4c0897f0fff958a81201, State: Running, Role: LEADER
I20251024 20:02:03.516489 23377 consensus_queue.cc:237] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:03.517066 23042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0b28aaca8d94b7abbe668af88781aaa" candidate_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b58eaa4961f4c0897f0fff958a81201" is_pre_election: true
I20251024 20:02:03.517302 23117 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0b28aaca8d94b7abbe668af88781aaa" candidate_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" is_pre_election: true
W20251024 20:02:03.517472 22930 leader_election.cc:343] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): Illegal state: cannot vote while shut down
I20251024 20:02:03.517472 22860 catalog_manager.cc:5649] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 reported cstate change: term changed from 1 to 2, leader changed from 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67) to 6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66). New cstate: current_term: 2 leader_uuid: "6b58eaa4961f4c0897f0fff958a81201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } } }
I20251024 20:02:03.517563 22930 leader_election.cc:304] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d7f551b0d494bf69275a3f42b1d7ba4; no voters: 11c499cd2fd34cb0aa6c2a8feadc73bf, 6b58eaa4961f4c0897f0fff958a81201
I20251024 20:02:03.517638 23374 raft_consensus.cc:2749] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 20:02:03.750072 22956 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:03.750366 23107 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:03.750500 23032 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 f51b793fdeca4f98b2e5d95363a18523 | 127.18.153.126:35797 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 11c499cd2fd34cb0aa6c2a8feadc73bf | 127.18.153.67:40675 | HEALTHY | <none>   |       0        |       0
 5d7f551b0d494bf69275a3f42b1d7ba4 | 127.18.153.65:39037 | HEALTHY | <none>   |       0        |       0
 6b58eaa4961f4c0897f0fff958a81201 | 127.18.153.66:36927 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet f0b28aaca8d94b7abbe668af88781aaa of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): not running
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
  5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037): RUNNING
  6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66:36927): RUNNING [LEADER]
All reported replicas are:
  A = 11c499cd2fd34cb0aa6c2a8feadc73bf
  B = 5d7f551b0d494bf69275a3f42b1d7ba4
  C = 6b58eaa4961f4c0897f0fff958a81201
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A   B   C*             |              |              | Yes
 A             | [config not available] |              |              | 
 B             | A   B   C              | 2            | -1           | Yes
 C             | A   B   C*             | 2            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:03.758503 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20251024 20:02:03.968947 23006 consensus_peers.cc:597] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 -> Peer 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): Couldn't send request to peer 11c499cd2fd34cb0aa6c2a8feadc73bf. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20251024 20:02:03.972913 22966 raft_consensus.cc:1275] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 2 FOLLOWER]: Refusing update from remote peer 6b58eaa4961f4c0897f0fff958a81201: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251024 20:02:03.973152 23373 consensus_queue.cc:1048] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251024 20:02:04.540283 22956 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:04.540531 23107 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:04.541461 23032 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 f51b793fdeca4f98b2e5d95363a18523 | 127.18.153.126:35797 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 11c499cd2fd34cb0aa6c2a8feadc73bf | 127.18.153.67:40675 | HEALTHY | <none>   |       0        |       0
 5d7f551b0d494bf69275a3f42b1d7ba4 | 127.18.153.65:39037 | HEALTHY | <none>   |       0        |       0
 6b58eaa4961f4c0897f0fff958a81201 | 127.18.153.66:36927 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet f0b28aaca8d94b7abbe668af88781aaa of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): not running
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
  5d7f551b0d494bf69275a3f42b1d7ba4 (127.18.153.65:39037): RUNNING
  6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66:36927): RUNNING [LEADER]
All reported replicas are:
  A = 11c499cd2fd34cb0aa6c2a8feadc73bf
  B = 5d7f551b0d494bf69275a3f42b1d7ba4
  C = 6b58eaa4961f4c0897f0fff958a81201
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A   B   C*             |              |              | Yes
 A             | [config not available] |              |              | 
 B             | A   B   C*             | 2            | -1           | Yes
 C             | A   B   C*             | 2            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:04.549733 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:04.591871 23438 raft_consensus.cc:1012] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201: Attempting to remove follower 11c499cd2fd34cb0aa6c2a8feadc73bf from the Raft config. Reason: The tablet replica hosted on peer 11c499cd2fd34cb0aa6c2a8feadc73bf has failed
I20251024 20:02:04.592136 23438 consensus_queue.cc:237] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } }
I20251024 20:02:04.592582 22966 raft_consensus.cc:1275] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 2 FOLLOWER]: Refusing update from remote peer 6b58eaa4961f4c0897f0fff958a81201: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251024 20:02:04.592803 23438 consensus_queue.cc:1048] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:02:04.593562 23437 raft_consensus.cc:2955] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } } }
I20251024 20:02:04.593672 22966 raft_consensus.cc:2955] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } } }
I20251024 20:02:04.594770 22860 catalog_manager.cc:5649] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 reported cstate change: config changed from index -1 to 3, VOTER 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67) evicted. New cstate: current_term: 2 leader_uuid: "6b58eaa4961f4c0897f0fff958a81201" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } } }
I20251024 20:02:04.596007 23107 tablet_service.cc:1552] Processing DeleteTablet for tablet f0b28aaca8d94b7abbe668af88781aaa with delete_type TABLET_DATA_TOMBSTONED (TS 11c499cd2fd34cb0aa6c2a8feadc73bf not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45484
I20251024 20:02:04.596155 23042 consensus_queue.cc:237] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } } peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } }
I20251024 20:02:04.596240 23444 ts_tablet_manager.cc:1916] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251024 20:02:04.596487 22966 raft_consensus.cc:1275] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 2 FOLLOWER]: Refusing update from remote peer 6b58eaa4961f4c0897f0fff958a81201: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20251024 20:02:04.596623 23006 consensus_peers.cc:597] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 -> Peer 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): Couldn't send request to peer 11c499cd2fd34cb0aa6c2a8feadc73bf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20251024 20:02:04.596639 23437 consensus_queue.cc:1048] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251024 20:02:04.597195 23438 raft_consensus.cc:2955] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } } peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } }
I20251024 20:02:04.597327 22966 raft_consensus.cc:2955] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } } peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } }
I20251024 20:02:04.597959 23444 ts_tablet_manager.cc:1929] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20251024 20:02:04.598088 23444 log.cc:1199] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: Deleting WAL directory at /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1761336053687453-19045-0/minicluster-data/ts-2-root/wals/f0b28aaca8d94b7abbe668af88781aaa
I20251024 20:02:04.598137 22847 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:VOTER RPC for tablet f0b28aaca8d94b7abbe668af88781aaa with cas_config_opid_index 3: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20251024 20:02:04.598246 22860 catalog_manager.cc:5649] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 reported cstate change: config changed from index 3 to 4, VOTER 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67) added. New cstate: current_term: 2 leader_uuid: "6b58eaa4961f4c0897f0fff958a81201" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } } peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } } }
I20251024 20:02:04.598691 22848 catalog_manager.cc:4985] TS 11c499cd2fd34cb0aa6c2a8feadc73bf (127.18.153.67:40675): tablet f0b28aaca8d94b7abbe668af88781aaa (table test-workload [id=fb4aa255e8e44fe5b565f2b0d2b39caf]) successfully deleted
I20251024 20:02:05.080430 23446 ts_tablet_manager.cc:933] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: Initiating tablet copy from peer 6b58eaa4961f4c0897f0fff958a81201 (127.18.153.66:36927)
I20251024 20:02:05.080631 23446 tablet_copy_client.cc:323] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: tablet copy: Beginning tablet copy session from remote peer at address 127.18.153.66:36927
I20251024 20:02:05.080919 23052 tablet_copy_service.cc:140] P 6b58eaa4961f4c0897f0fff958a81201: Received BeginTabletCopySession request for tablet f0b28aaca8d94b7abbe668af88781aaa from peer 11c499cd2fd34cb0aa6c2a8feadc73bf ({username='slave'} at 127.0.0.1:60086)
I20251024 20:02:05.080996 23052 tablet_copy_service.cc:161] P 6b58eaa4961f4c0897f0fff958a81201: Beginning new tablet copy session on tablet f0b28aaca8d94b7abbe668af88781aaa from peer 11c499cd2fd34cb0aa6c2a8feadc73bf at {username='slave'} at 127.0.0.1:60086: session id = 11c499cd2fd34cb0aa6c2a8feadc73bf-f0b28aaca8d94b7abbe668af88781aaa
I20251024 20:02:05.081517 23052 tablet_copy_source_session.cc:215] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201: Tablet Copy: opened 0 blocks and 1 log segments
I20251024 20:02:05.081827 23446 ts_tablet_manager.cc:1916] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: Deleting tablet data with delete state TABLET_DATA_COPYING
I20251024 20:02:05.083441 23446 ts_tablet_manager.cc:1929] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20251024 20:02:05.083532 23446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0b28aaca8d94b7abbe668af88781aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:05.084080 23446 tablet_copy_client.cc:806] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: tablet copy: Starting download of 0 data blocks...
I20251024 20:02:05.084189 23446 tablet_copy_client.cc:670] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: tablet copy: Starting download of 1 WAL segments...
I20251024 20:02:05.084775 23446 tablet_copy_client.cc:538] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251024 20:02:05.085310 23446 tablet_bootstrap.cc:492] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: Bootstrap starting.
I20251024 20:02:05.086552 23446 tablet_bootstrap.cc:492] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251024 20:02:05.086720 23446 tablet_bootstrap.cc:492] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: Bootstrap complete.
I20251024 20:02:05.086814 23446 ts_tablet_manager.cc:1403] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: Time spent bootstrapping tablet: real 0.002s	user 0.000s	sys 0.001s
I20251024 20:02:05.086956 23446 raft_consensus.cc:359] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } } peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } }
I20251024 20:02:05.087033 23446 raft_consensus.cc:740] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c499cd2fd34cb0aa6c2a8feadc73bf, State: Initialized, Role: FOLLOWER
I20251024 20:02:05.087096 23446 consensus_queue.cc:260] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5d7f551b0d494bf69275a3f42b1d7ba4" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39037 } } peers { permanent_uuid: "6b58eaa4961f4c0897f0fff958a81201" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 36927 } } peers { permanent_uuid: "11c499cd2fd34cb0aa6c2a8feadc73bf" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 40675 } }
I20251024 20:02:05.087257 23446 ts_tablet_manager.cc:1434] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: Time spent starting tablet: real 0.000s	user 0.000s	sys 0.001s
I20251024 20:02:05.087531 23052 tablet_copy_service.cc:342] P 6b58eaa4961f4c0897f0fff958a81201: Request end of tablet copy session 11c499cd2fd34cb0aa6c2a8feadc73bf-f0b28aaca8d94b7abbe668af88781aaa received from {username='slave'} at 127.0.0.1:60086
I20251024 20:02:05.087591 23052 tablet_copy_service.cc:434] P 6b58eaa4961f4c0897f0fff958a81201: ending tablet copy session 11c499cd2fd34cb0aa6c2a8feadc73bf-f0b28aaca8d94b7abbe668af88781aaa on tablet f0b28aaca8d94b7abbe668af88781aaa with peer 11c499cd2fd34cb0aa6c2a8feadc73bf
I20251024 20:02:05.537246 23117 raft_consensus.cc:1217] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.3->[2.4-2.4]   Dedup: 2.4->[]
I20251024 20:02:05.573268 23107 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:05.573535 22956 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:05.573953 23032 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 f51b793fdeca4f98b2e5d95363a18523 | 127.18.153.126:35797 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 11c499cd2fd34cb0aa6c2a8feadc73bf | 127.18.153.67:40675 | HEALTHY | <none>   |       0        |       0
 5d7f551b0d494bf69275a3f42b1d7ba4 | 127.18.153.65:39037 | HEALTHY | <none>   |       0        |       0
 6b58eaa4961f4c0897f0fff958a81201 | 127.18.153.66:36927 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
     Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20251024 20:02:05.582252 19045 log_verifier.cc:126] Checking tablet f0b28aaca8d94b7abbe668af88781aaa
I20251024 20:02:05.594367 19045 log_verifier.cc:177] Verified matching terms for 4 ops in tablet f0b28aaca8d94b7abbe668af88781aaa
I20251024 20:02:05.595307 19045 tablet_server.cc:178] TabletServer@127.18.153.65:0 shutting down...
I20251024 20:02:05.598302 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:05.598438 19045 tablet_replica.cc:333] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4: stopping tablet replica
I20251024 20:02:05.598493 19045 raft_consensus.cc:2243] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251024 20:02:05.598655 19045 raft_consensus.cc:2272] T f0b28aaca8d94b7abbe668af88781aaa P 5d7f551b0d494bf69275a3f42b1d7ba4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:05.611775 19045 tablet_server.cc:195] TabletServer@127.18.153.65:0 shutdown complete.
I20251024 20:02:05.612764 19045 tablet_server.cc:178] TabletServer@127.18.153.66:0 shutting down...
I20251024 20:02:05.615743 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:05.615868 19045 tablet_replica.cc:333] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201: stopping tablet replica
I20251024 20:02:05.615947 19045 raft_consensus.cc:2243] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 2 LEADER]: Raft consensus shutting down.
I20251024 20:02:05.616024 19045 raft_consensus.cc:2272] T f0b28aaca8d94b7abbe668af88781aaa P 6b58eaa4961f4c0897f0fff958a81201 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:05.628638 19045 tablet_server.cc:195] TabletServer@127.18.153.66:0 shutdown complete.
I20251024 20:02:05.629479 19045 tablet_server.cc:178] TabletServer@127.18.153.67:0 shutting down...
I20251024 20:02:05.632154 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:05.632262 19045 tablet_replica.cc:333] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf: stopping tablet replica
I20251024 20:02:05.632329 19045 raft_consensus.cc:2243] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 2 FOLLOWER]: Raft consensus shutting down.
I20251024 20:02:05.632373 19045 raft_consensus.cc:2272] T f0b28aaca8d94b7abbe668af88781aaa P 11c499cd2fd34cb0aa6c2a8feadc73bf [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:05.644955 19045 tablet_server.cc:195] TabletServer@127.18.153.67:0 shutdown complete.
I20251024 20:02:05.645874 19045 master.cc:561] Master@127.18.153.126:35797 shutting down...
I20251024 20:02:05.648584 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:05.648662 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:05.648684 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P f51b793fdeca4f98b2e5d95363a18523: stopping tablet replica
I20251024 20:02:05.660415 19045 master.cc:583] Master@127.18.153.126:35797 shutdown complete.
[       OK ] FailedTabletsAreReplacedITest.OneReplica/0 (4570 ms)
[ RUN      ] FailedTabletsAreReplacedITest.OneReplica/1
I20251024 20:02:05.665138 19045 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.153.126:44001
I20251024 20:02:05.665304 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:05.666450 23482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:05.666467 23481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:05.666586 19045 server_base.cc:1047] running on GCE node
W20251024 20:02:05.666498 23484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:05.666855 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:05.666896 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:05.666922 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336125666921 us; error 0 us; skew 500 ppm
I20251024 20:02:05.667507 19045 webserver.cc:492] Webserver started at http://127.18.153.126:37477/ using document root <none> and password file <none>
I20251024 20:02:05.667582 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:05.667625 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:05.667677 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:05.667939 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/master-0-root/instance:
uuid: "91a71ba4ec254020985ba241455aceef"
format_stamp: "Formatted at 2025-10-24 20:02:05 on dist-test-slave-375r"
I20251024 20:02:05.668816 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:02:05.669284 23489 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.669418 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.669458 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/master-0-root
uuid: "91a71ba4ec254020985ba241455aceef"
format_stamp: "Formatted at 2025-10-24 20:02:05 on dist-test-slave-375r"
I20251024 20:02:05.669505 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:05.681712 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:05.681926 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:05.684638 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.126:44001
I20251024 20:02:05.687103 23542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.126:44001 every 8 connection(s)
I20251024 20:02:05.687263 23543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:05.688329 23543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef: Bootstrap starting.
I20251024 20:02:05.688632 23543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:05.689147 23543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef: No bootstrap required, opened a new log
I20251024 20:02:05.689280 23543 raft_consensus.cc:359] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a71ba4ec254020985ba241455aceef" member_type: VOTER }
I20251024 20:02:05.689335 23543 raft_consensus.cc:385] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:05.689349 23543 raft_consensus.cc:740] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a71ba4ec254020985ba241455aceef, State: Initialized, Role: FOLLOWER
I20251024 20:02:05.689404 23543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a71ba4ec254020985ba241455aceef" member_type: VOTER }
I20251024 20:02:05.689438 23543 raft_consensus.cc:399] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 20:02:05.689464 23543 raft_consensus.cc:493] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 20:02:05.689494 23543 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:05.689977 23543 raft_consensus.cc:515] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a71ba4ec254020985ba241455aceef" member_type: VOTER }
I20251024 20:02:05.690039 23543 leader_election.cc:304] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91a71ba4ec254020985ba241455aceef; no voters: 
I20251024 20:02:05.690160 23543 leader_election.cc:290] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [CANDIDATE]: Term 1 election: Requested vote from peers 
I20251024 20:02:05.690207 23546 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:02:05.690310 23543 sys_catalog.cc:565] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [sys.catalog]: configured and running, proceeding with master startup.
I20251024 20:02:05.690354 23546 raft_consensus.cc:697] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [term 1 LEADER]: Becoming Leader. State: Replica: 91a71ba4ec254020985ba241455aceef, State: Running, Role: LEADER
I20251024 20:02:05.690431 23546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a71ba4ec254020985ba241455aceef" member_type: VOTER }
I20251024 20:02:05.690644 23548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91a71ba4ec254020985ba241455aceef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a71ba4ec254020985ba241455aceef" member_type: VOTER } }
I20251024 20:02:05.690649 23547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91a71ba4ec254020985ba241455aceef. Latest consensus state: current_term: 1 leader_uuid: "91a71ba4ec254020985ba241455aceef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a71ba4ec254020985ba241455aceef" member_type: VOTER } }
I20251024 20:02:05.690723 23548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [sys.catalog]: This master's current role is: LEADER
I20251024 20:02:05.690858 23547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [sys.catalog]: This master's current role is: LEADER
I20251024 20:02:05.690923 23552 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 20:02:05.691057 23552 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 20:02:05.691666 23552 catalog_manager.cc:1357] Generated new cluster ID: a3ccc33be7944f148f7e887757a27c85
I20251024 20:02:05.691722 23552 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 20:02:05.691742 19045 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251024 20:02:05.701403 23552 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 20:02:05.701838 23552 catalog_manager.cc:1514] Loading token signing keys...
I20251024 20:02:05.705111 23552 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef: Generated new TSK 0
I20251024 20:02:05.705199 23552 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 20:02:05.707489 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:05.708743 23567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:05.708719 23564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:05.708853 19045 server_base.cc:1047] running on GCE node
W20251024 20:02:05.708864 23565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:05.709069 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:05.709108 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:05.709131 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336125709132 us; error 0 us; skew 500 ppm
I20251024 20:02:05.709721 19045 webserver.cc:492] Webserver started at http://127.18.153.65:37521/ using document root <none> and password file <none>
I20251024 20:02:05.709806 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:05.709856 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:05.709913 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:05.710201 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-0-root/instance:
uuid: "1b9eef424e124e899ca2c6324f2473e9"
format_stamp: "Formatted at 2025-10-24 20:02:05 on dist-test-slave-375r"
I20251024 20:02:05.711159 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.001s
I20251024 20:02:05.711637 23572 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.711777 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.711822 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-0-root
uuid: "1b9eef424e124e899ca2c6324f2473e9"
format_stamp: "Formatted at 2025-10-24 20:02:05 on dist-test-slave-375r"
I20251024 20:02:05.711874 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:05.729804 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:05.730026 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:05.730203 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:02:05.730414 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:02:05.730445 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.730481 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:02:05.730513 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.733438 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.65:39289
I20251024 20:02:05.733472 23635 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.65:39289 every 8 connection(s)
I20251024 20:02:05.733815 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:05.734845 23640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:05.734917 23641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:05.735121 19045 server_base.cc:1047] running on GCE node
W20251024 20:02:05.735860 23643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:05.736032 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:05.736068 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:05.736080 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336125736081 us; error 0 us; skew 500 ppm
I20251024 20:02:05.736753 19045 webserver.cc:492] Webserver started at http://127.18.153.66:42527/ using document root <none> and password file <none>
I20251024 20:02:05.736845 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:05.736901 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:05.736958 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:05.737345 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-1-root/instance:
uuid: "9a33a847488d4df9b881793fd2bea9af"
format_stamp: "Formatted at 2025-10-24 20:02:05 on dist-test-slave-375r"
I20251024 20:02:05.737426 23636 heartbeater.cc:344] Connected to a master server at 127.18.153.126:44001
I20251024 20:02:05.737483 23636 heartbeater.cc:461] Registering TS with master...
I20251024 20:02:05.737605 23636 heartbeater.cc:507] Master 127.18.153.126:44001 requested a full tablet report, sending...
I20251024 20:02:05.737833 23506 ts_manager.cc:194] Registered new tserver with Master: 1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289)
I20251024 20:02:05.738410 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:05.738444 23506 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50778
I20251024 20:02:05.738969 23648 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.739100 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.001s	sys 0.000s
I20251024 20:02:05.739148 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-1-root
uuid: "9a33a847488d4df9b881793fd2bea9af"
format_stamp: "Formatted at 2025-10-24 20:02:05 on dist-test-slave-375r"
I20251024 20:02:05.739195 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:05.744756 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:05.744951 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:05.745114 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:02:05.745314 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:02:05.745342 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.745378 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:02:05.745404 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.748405 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.66:35565
I20251024 20:02:05.748466 23711 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.66:35565 every 8 connection(s)
I20251024 20:02:05.748793 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:05.749927 23716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:05.750159 23715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:05.750185 19045 server_base.cc:1047] running on GCE node
W20251024 20:02:05.750355 23718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:05.750535 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:05.750581 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:05.750599 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336125750599 us; error 0 us; skew 500 ppm
I20251024 20:02:05.751258 19045 webserver.cc:492] Webserver started at http://127.18.153.67:39751/ using document root <none> and password file <none>
I20251024 20:02:05.751356 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:05.751451 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:05.751503 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:05.751845 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-2-root/instance:
uuid: "3d3ae51c29f44826882a9a53da3bc61b"
format_stamp: "Formatted at 2025-10-24 20:02:05 on dist-test-slave-375r"
I20251024 20:02:05.751878 23712 heartbeater.cc:344] Connected to a master server at 127.18.153.126:44001
I20251024 20:02:05.751925 23712 heartbeater.cc:461] Registering TS with master...
I20251024 20:02:05.752023 23712 heartbeater.cc:507] Master 127.18.153.126:44001 requested a full tablet report, sending...
I20251024 20:02:05.752276 23506 ts_manager.cc:194] Registered new tserver with Master: 9a33a847488d4df9b881793fd2bea9af (127.18.153.66:35565)
I20251024 20:02:05.752750 23506 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50794
I20251024 20:02:05.752929 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:02:05.753396 23723 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.753500 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.753541 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-2-root
uuid: "3d3ae51c29f44826882a9a53da3bc61b"
format_stamp: "Formatted at 2025-10-24 20:02:05 on dist-test-slave-375r"
I20251024 20:02:05.753593 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:05.768100 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:05.768373 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:05.768558 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:02:05.768766 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:02:05.768793 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.768831 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:02:05.768857 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.772339 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.67:45845
I20251024 20:02:05.772389 23786 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.67:45845 every 8 connection(s)
I20251024 20:02:05.772720 19045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 20:02:05.773937 23791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:05.773955 23790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 20:02:05.774045 23793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 20:02:05.774212 19045 server_base.cc:1047] running on GCE node
I20251024 20:02:05.774346 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251024 20:02:05.774379 19045 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251024 20:02:05.774396 19045 hybrid_clock.cc:648] HybridClock initialized: now 1761336125774397 us; error 0 us; skew 500 ppm
I20251024 20:02:05.775071 19045 webserver.cc:492] Webserver started at http://127.18.153.68:33953/ using document root <none> and password file <none>
I20251024 20:02:05.775156 19045 fs_manager.cc:362] Metadata directory not provided
I20251024 20:02:05.775211 19045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 20:02:05.775301 19045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 20:02:05.775648 19045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-3-root/instance:
uuid: "56050fa24271490097d19b69be5da104"
format_stamp: "Formatted at 2025-10-24 20:02:05 on dist-test-slave-375r"
I20251024 20:02:05.776304 23787 heartbeater.cc:344] Connected to a master server at 127.18.153.126:44001
I20251024 20:02:05.776378 23787 heartbeater.cc:461] Registering TS with master...
I20251024 20:02:05.776515 23787 heartbeater.cc:507] Master 127.18.153.126:44001 requested a full tablet report, sending...
I20251024 20:02:05.776734 23506 ts_manager.cc:194] Registered new tserver with Master: 3d3ae51c29f44826882a9a53da3bc61b (127.18.153.67:45845)
I20251024 20:02:05.777068 23506 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50810
I20251024 20:02:05.777309 19045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
I20251024 20:02:05.777859 23798 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.777977 19045 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.001s
I20251024 20:02:05.778023 19045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-3-root
uuid: "56050fa24271490097d19b69be5da104"
format_stamp: "Formatted at 2025-10-24 20:02:05 on dist-test-slave-375r"
I20251024 20:02:05.778079 19045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 20:02:05.794438 19045 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20251024 20:02:05.794723 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 20:02:05.794893 19045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 20:02:05.795090 19045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 20:02:05.795133 19045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.795171 19045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 20:02:05.795197 19045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20251024 20:02:05.798544 19045 rpc_server.cc:307] RPC server started. Bound to: 127.18.153.68:34307
I20251024 20:02:05.798617 23861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.153.68:34307 every 8 connection(s)
I20251024 20:02:05.801694 23862 heartbeater.cc:344] Connected to a master server at 127.18.153.126:44001
I20251024 20:02:05.801757 23862 heartbeater.cc:461] Registering TS with master...
I20251024 20:02:05.801862 23862 heartbeater.cc:507] Master 127.18.153.126:44001 requested a full tablet report, sending...
I20251024 20:02:05.802096 23506 ts_manager.cc:194] Registered new tserver with Master: 56050fa24271490097d19b69be5da104 (127.18.153.68:34307)
I20251024 20:02:05.802415 23506 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50820
I20251024 20:02:05.803124 19045 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.004385631s
I20251024 20:02:05.807374 19045 test_util.cc:276] Using random seed: 102488845
I20251024 20:02:05.811344 23506 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50832:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20251024 20:02:05.816135 23751 tablet_service.cc:1505] Processing CreateTablet for tablet 9624e8d395694aa589a3c0e956e8a2d9 (DEFAULT_TABLE table=test-workload [id=5ef79921f748446f9cae1b44ce4b1df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 20:02:05.816301 23751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9624e8d395694aa589a3c0e956e8a2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:05.816385 23600 tablet_service.cc:1505] Processing CreateTablet for tablet 9624e8d395694aa589a3c0e956e8a2d9 (DEFAULT_TABLE table=test-workload [id=5ef79921f748446f9cae1b44ce4b1df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 20:02:05.816481 23600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9624e8d395694aa589a3c0e956e8a2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:05.817472 23882 tablet_bootstrap.cc:492] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b: Bootstrap starting.
I20251024 20:02:05.817704 23883 tablet_bootstrap.cc:492] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9: Bootstrap starting.
I20251024 20:02:05.817867 23676 tablet_service.cc:1505] Processing CreateTablet for tablet 9624e8d395694aa589a3c0e956e8a2d9 (DEFAULT_TABLE table=test-workload [id=5ef79921f748446f9cae1b44ce4b1df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 20:02:05.817967 23676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9624e8d395694aa589a3c0e956e8a2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 20:02:05.817994 23882 tablet_bootstrap.cc:654] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:05.818217 23883 tablet_bootstrap.cc:654] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:05.818769 23883 tablet_bootstrap.cc:492] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9: No bootstrap required, opened a new log
I20251024 20:02:05.818815 23883 ts_tablet_manager.cc:1403] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9: Time spent bootstrapping tablet: real 0.001s	user 0.001s	sys 0.000s
I20251024 20:02:05.818989 23883 raft_consensus.cc:359] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } }
I20251024 20:02:05.819061 23883 raft_consensus.cc:385] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:05.819087 23883 raft_consensus.cc:740] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b9eef424e124e899ca2c6324f2473e9, State: Initialized, Role: FOLLOWER
I20251024 20:02:05.819149 23886 tablet_bootstrap.cc:492] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af: Bootstrap starting.
I20251024 20:02:05.819218 23883 consensus_queue.cc:260] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } }
I20251024 20:02:05.819444 23636 heartbeater.cc:499] Master 127.18.153.126:44001 was elected leader, sending a full tablet report...
I20251024 20:02:05.819494 23882 tablet_bootstrap.cc:492] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b: No bootstrap required, opened a new log
I20251024 20:02:05.819552 23882 ts_tablet_manager.cc:1403] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I20251024 20:02:05.819434 23883 ts_tablet_manager.cc:1434] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20251024 20:02:05.819701 23886 tablet_bootstrap.cc:654] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af: Neither blocks nor log segments found. Creating new log.
I20251024 20:02:05.819720 23882 raft_consensus.cc:359] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } }
I20251024 20:02:05.819824 23882 raft_consensus.cc:385] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:05.819852 23882 raft_consensus.cc:740] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3ae51c29f44826882a9a53da3bc61b, State: Initialized, Role: FOLLOWER
I20251024 20:02:05.819906 23882 consensus_queue.cc:260] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } }
I20251024 20:02:05.820067 23882 ts_tablet_manager.cc:1434] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b: Time spent starting tablet: real 0.000s	user 0.000s	sys 0.001s
I20251024 20:02:05.820183 23787 heartbeater.cc:499] Master 127.18.153.126:44001 was elected leader, sending a full tablet report...
I20251024 20:02:05.820601 23886 tablet_bootstrap.cc:492] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af: No bootstrap required, opened a new log
I20251024 20:02:05.820657 23886 ts_tablet_manager.cc:1403] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I20251024 20:02:05.820808 23886 raft_consensus.cc:359] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } }
I20251024 20:02:05.820899 23886 raft_consensus.cc:385] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 20:02:05.820922 23886 raft_consensus.cc:740] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a33a847488d4df9b881793fd2bea9af, State: Initialized, Role: FOLLOWER
I20251024 20:02:05.820989 23886 consensus_queue.cc:260] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } }
I20251024 20:02:05.821144 23886 ts_tablet_manager.cc:1434] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af: Time spent starting tablet: real 0.000s	user 0.001s	sys 0.000s
I20251024 20:02:05.821173 23712 heartbeater.cc:499] Master 127.18.153.126:44001 was elected leader, sending a full tablet report...
I20251024 20:02:05.947568 23887 raft_consensus.cc:493] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 20:02:05.947688 23887 raft_consensus.cc:515] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } }
I20251024 20:02:05.947873 23887 leader_election.cc:290] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d3ae51c29f44826882a9a53da3bc61b (127.18.153.67:45845), 9a33a847488d4df9b881793fd2bea9af (127.18.153.66:35565)
I20251024 20:02:05.951524 23686 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9624e8d395694aa589a3c0e956e8a2d9" candidate_uuid: "1b9eef424e124e899ca2c6324f2473e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a33a847488d4df9b881793fd2bea9af" is_pre_election: true
I20251024 20:02:05.951646 23686 raft_consensus.cc:2468] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b9eef424e124e899ca2c6324f2473e9 in term 0.
I20251024 20:02:05.951526 23761 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9624e8d395694aa589a3c0e956e8a2d9" candidate_uuid: "1b9eef424e124e899ca2c6324f2473e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3ae51c29f44826882a9a53da3bc61b" is_pre_election: true
I20251024 20:02:05.951692 23761 raft_consensus.cc:2468] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b9eef424e124e899ca2c6324f2473e9 in term 0.
I20251024 20:02:05.951865 23573 leader_election.cc:304] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9eef424e124e899ca2c6324f2473e9, 3d3ae51c29f44826882a9a53da3bc61b; no voters: 
I20251024 20:02:05.951998 23887 raft_consensus.cc:2804] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 20:02:05.952044 23887 raft_consensus.cc:493] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 20:02:05.952071 23887 raft_consensus.cc:3060] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:05.952736 23887 raft_consensus.cc:515] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } }
I20251024 20:02:05.952872 23887 leader_election.cc:290] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [CANDIDATE]: Term 1 election: Requested vote from peers 3d3ae51c29f44826882a9a53da3bc61b (127.18.153.67:45845), 9a33a847488d4df9b881793fd2bea9af (127.18.153.66:35565)
I20251024 20:02:05.953073 23761 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9624e8d395694aa589a3c0e956e8a2d9" candidate_uuid: "1b9eef424e124e899ca2c6324f2473e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3ae51c29f44826882a9a53da3bc61b"
I20251024 20:02:05.953133 23761 raft_consensus.cc:3060] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:05.953128 23686 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9624e8d395694aa589a3c0e956e8a2d9" candidate_uuid: "1b9eef424e124e899ca2c6324f2473e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a33a847488d4df9b881793fd2bea9af"
I20251024 20:02:05.953231 23686 raft_consensus.cc:3060] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 0 FOLLOWER]: Advancing to term 1
I20251024 20:02:05.953794 23761 raft_consensus.cc:2468] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b9eef424e124e899ca2c6324f2473e9 in term 1.
I20251024 20:02:05.953888 23686 raft_consensus.cc:2468] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b9eef424e124e899ca2c6324f2473e9 in term 1.
I20251024 20:02:05.953994 23573 leader_election.cc:304] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9eef424e124e899ca2c6324f2473e9, 3d3ae51c29f44826882a9a53da3bc61b; no voters: 
I20251024 20:02:05.954119 23887 raft_consensus.cc:2804] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 20:02:05.954275 23887 raft_consensus.cc:697] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [term 1 LEADER]: Becoming Leader. State: Replica: 1b9eef424e124e899ca2c6324f2473e9, State: Running, Role: LEADER
I20251024 20:02:05.954329 23887 consensus_queue.cc:237] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } }
I20251024 20:02:05.954929 23505 catalog_manager.cc:5649] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65). New cstate: current_term: 1 leader_uuid: "1b9eef424e124e899ca2c6324f2473e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } health_report { overall_health: HEALTHY } } }
I20251024 20:02:05.984002 23751 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:05.984823 23600 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20251024 20:02:05.986333 23826 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:05.988034 23676 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 91a71ba4ec254020985ba241455aceef | 127.18.153.126:44001 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1b9eef424e124e899ca2c6324f2473e9 | 127.18.153.65:39289 | HEALTHY | <none>   |       1        |       0
 3d3ae51c29f44826882a9a53da3bc61b | 127.18.153.67:45845 | HEALTHY | <none>   |       0        |       0
 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | HEALTHY | <none>   |       0        |       0
 9a33a847488d4df9b881793fd2bea9af | 127.18.153.66:35565 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
Tablet 9624e8d395694aa589a3c0e956e8a2d9 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
  3d3ae51c29f44826882a9a53da3bc61b (127.18.153.67:45845): RUNNING
  9a33a847488d4df9b881793fd2bea9af (127.18.153.66:35565): RUNNING
  1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289): RUNNING [LEADER]
All reported replicas are:
  A = 3d3ae51c29f44826882a9a53da3bc61b
  B = 9a33a847488d4df9b881793fd2bea9af
  C = 1b9eef424e124e899ca2c6324f2473e9
The consensus matrix is:
 Config source |   Replicas   | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A   B   C*   |              |              | Yes
 A             | A   B   C    | 1            | -1           | Yes
 B             | A   B   C    | 1            | -1           | Yes
 C             | A   B   C*   | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |       Status       | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | CONSENSUS_MISMATCH | 1             | 0       | 0          | 0                | 1

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |        Host         | Replica Count
-------+----------------------------------+---------------------+---------------
 Small | 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:05.997135 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:06.170544 23751 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:06.170559 23826 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:06.170668 23600 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20251024 20:02:06.172353 23676 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 91a71ba4ec254020985ba241455aceef | 127.18.153.126:44001 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1b9eef424e124e899ca2c6324f2473e9 | 127.18.153.65:39289 | HEALTHY | <none>   |       1        |       0
 3d3ae51c29f44826882a9a53da3bc61b | 127.18.153.67:45845 | HEALTHY | <none>   |       0        |       0
 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | HEALTHY | <none>   |       0        |       0
 9a33a847488d4df9b881793fd2bea9af | 127.18.153.66:35565 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
Tablet 9624e8d395694aa589a3c0e956e8a2d9 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
  3d3ae51c29f44826882a9a53da3bc61b (127.18.153.67:45845): RUNNING
  9a33a847488d4df9b881793fd2bea9af (127.18.153.66:35565): RUNNING
  1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289): RUNNING [LEADER]
All reported replicas are:
  A = 3d3ae51c29f44826882a9a53da3bc61b
  B = 9a33a847488d4df9b881793fd2bea9af
  C = 1b9eef424e124e899ca2c6324f2473e9
The consensus matrix is:
 Config source |   Replicas   | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A   B   C*   |              |              | Yes
 A             | A   B   C    | 1            | -1           | Yes
 B             | A   B   C    | 1            | -1           | Yes
 C             | A   B   C*   | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |       Status       | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | CONSENSUS_MISMATCH | 1             | 0       | 0          | 0                | 1

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |        Host         | Replica Count
-------+----------------------------------+---------------------+---------------
 Small | 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:06.182204 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:06.431516 23751 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:06.431581 23676 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:06.431588 23600 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20251024 20:02:06.433655 23826 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 91a71ba4ec254020985ba241455aceef | 127.18.153.126:44001 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1b9eef424e124e899ca2c6324f2473e9 | 127.18.153.65:39289 | HEALTHY | <none>   |       1        |       0
 3d3ae51c29f44826882a9a53da3bc61b | 127.18.153.67:45845 | HEALTHY | <none>   |       0        |       0
 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | HEALTHY | <none>   |       0        |       0
 9a33a847488d4df9b881793fd2bea9af | 127.18.153.66:35565 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
Tablet 9624e8d395694aa589a3c0e956e8a2d9 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
  3d3ae51c29f44826882a9a53da3bc61b (127.18.153.67:45845): RUNNING
  9a33a847488d4df9b881793fd2bea9af (127.18.153.66:35565): RUNNING
  1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289): RUNNING [LEADER]
All reported replicas are:
  A = 3d3ae51c29f44826882a9a53da3bc61b
  B = 9a33a847488d4df9b881793fd2bea9af
  C = 1b9eef424e124e899ca2c6324f2473e9
The consensus matrix is:
 Config source |   Replicas   | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A   B   C*   |              |              | Yes
 A             | A   B   C    | 1            | -1           | Yes
 B             | A   B   C    | 1            | -1           | Yes
 C             | A   B   C*   | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |       Status       | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | CONSENSUS_MISMATCH | 1             | 0       | 0          | 0                | 1

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |        Host         | Replica Count
-------+----------------------------------+---------------------+---------------
 Small | 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:06.444492 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:06.476907 23974 consensus_queue.cc:1048] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:02:06.478540 23977 consensus_queue.cc:1048] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 20:02:06.802853 23862 heartbeater.cc:499] Master 127.18.153.126:44001 was elected leader, sending a full tablet report...
I20251024 20:02:06.806057 23751 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:06.806733 23826 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:06.806916 23600 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20251024 20:02:06.807271 23676 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 91a71ba4ec254020985ba241455aceef | 127.18.153.126:44001 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1b9eef424e124e899ca2c6324f2473e9 | 127.18.153.65:39289 | HEALTHY | <none>   |       1        |       0
 3d3ae51c29f44826882a9a53da3bc61b | 127.18.153.67:45845 | HEALTHY | <none>   |       0        |       0
 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | HEALTHY | <none>   |       0        |       0
 9a33a847488d4df9b881793fd2bea9af | 127.18.153.66:35565 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
     Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |        Host         | Replica Count
-------+----------------------------------+---------------------+---------------
 Small | 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20251024 20:02:06.817062 19045 log_verifier.cc:126] Checking tablet 9624e8d395694aa589a3c0e956e8a2d9
I20251024 20:02:06.834270 19045 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 9624e8d395694aa589a3c0e956e8a2d9
I20251024 20:02:06.834407 19045 tablet_replica.cc:333] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9: stopping tablet replica
I20251024 20:02:06.834455 19045 raft_consensus.cc:2243] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:06.834543 19045 raft_consensus.cc:2272] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:06.857957 23600 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:06.858212 23751 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:06.859334 23826 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:06.859578 23676 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 91a71ba4ec254020985ba241455aceef | 127.18.153.126:44001 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1b9eef424e124e899ca2c6324f2473e9 | 127.18.153.65:39289 | HEALTHY | <none>   |       0        |       0
 3d3ae51c29f44826882a9a53da3bc61b | 127.18.153.67:45845 | HEALTHY | <none>   |       0        |       0
 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | HEALTHY | <none>   |       0        |       0
 9a33a847488d4df9b881793fd2bea9af | 127.18.153.66:35565 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
Tablet 9624e8d395694aa589a3c0e956e8a2d9 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  3d3ae51c29f44826882a9a53da3bc61b (127.18.153.67:45845): RUNNING
  9a33a847488d4df9b881793fd2bea9af (127.18.153.66:35565): RUNNING
  1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
  A = 3d3ae51c29f44826882a9a53da3bc61b
  B = 9a33a847488d4df9b881793fd2bea9af
  C = 1b9eef424e124e899ca2c6324f2473e9
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A   B   C*             |              |              | Yes
 A             | A   B   C*             | 1            | -1           | Yes
 B             | A   B   C*             | 1            | -1           | Yes
 C             | [config not available] |              |              | 

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |        Host         | Replica Count
-------+----------------------------------+---------------------+---------------
 Small | 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:06.869772 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:07.043826 23600 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:07.044203 23751 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:07.044394 23676 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:07.045406 23826 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 91a71ba4ec254020985ba241455aceef | 127.18.153.126:44001 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1b9eef424e124e899ca2c6324f2473e9 | 127.18.153.65:39289 | HEALTHY | <none>   |       0        |       0
 3d3ae51c29f44826882a9a53da3bc61b | 127.18.153.67:45845 | HEALTHY | <none>   |       0        |       0
 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | HEALTHY | <none>   |       0        |       0
 9a33a847488d4df9b881793fd2bea9af | 127.18.153.66:35565 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
Tablet 9624e8d395694aa589a3c0e956e8a2d9 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  3d3ae51c29f44826882a9a53da3bc61b (127.18.153.67:45845): RUNNING
  9a33a847488d4df9b881793fd2bea9af (127.18.153.66:35565): RUNNING
  1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
  A = 3d3ae51c29f44826882a9a53da3bc61b
  B = 9a33a847488d4df9b881793fd2bea9af
  C = 1b9eef424e124e899ca2c6324f2473e9
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A   B   C*             |              |              | Yes
 A             | A   B   C*             | 1            | -1           | Yes
 B             | A   B   C*             | 1            | -1           | Yes
 C             | [config not available] |              |              | 

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |        Host         | Replica Count
-------+----------------------------------+---------------------+---------------
 Small | 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:07.054260 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:07.303669 23751 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:07.304029 23676 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:07.304277 23600 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:07.304780 23826 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 91a71ba4ec254020985ba241455aceef | 127.18.153.126:44001 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1b9eef424e124e899ca2c6324f2473e9 | 127.18.153.65:39289 | HEALTHY | <none>   |       0        |       0
 3d3ae51c29f44826882a9a53da3bc61b | 127.18.153.67:45845 | HEALTHY | <none>   |       0        |       0
 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | HEALTHY | <none>   |       0        |       0
 9a33a847488d4df9b881793fd2bea9af | 127.18.153.66:35565 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
Tablet 9624e8d395694aa589a3c0e956e8a2d9 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  3d3ae51c29f44826882a9a53da3bc61b (127.18.153.67:45845): RUNNING
  9a33a847488d4df9b881793fd2bea9af (127.18.153.66:35565): RUNNING
  1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
  A = 3d3ae51c29f44826882a9a53da3bc61b
  B = 9a33a847488d4df9b881793fd2bea9af
  C = 1b9eef424e124e899ca2c6324f2473e9
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A   B   C*             |              |              | Yes
 A             | A   B   C*             | 1            | -1           | Yes
 B             | A   B   C*             | 1            | -1           | Yes
 C             | [config not available] |              |              | 

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |        Host         | Replica Count
-------+----------------------------------+---------------------+---------------
 Small | 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:07.315027 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:07.676524 23751 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:07.676780 23600 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:07.677182 23676 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:07.680204 23826 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 91a71ba4ec254020985ba241455aceef | 127.18.153.126:44001 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1b9eef424e124e899ca2c6324f2473e9 | 127.18.153.65:39289 | HEALTHY | <none>   |       0        |       0
 3d3ae51c29f44826882a9a53da3bc61b | 127.18.153.67:45845 | HEALTHY | <none>   |       0        |       0
 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | HEALTHY | <none>   |       0        |       0
 9a33a847488d4df9b881793fd2bea9af | 127.18.153.66:35565 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
Tablet 9624e8d395694aa589a3c0e956e8a2d9 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  3d3ae51c29f44826882a9a53da3bc61b (127.18.153.67:45845): RUNNING
  9a33a847488d4df9b881793fd2bea9af (127.18.153.66:35565): RUNNING
  1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
  A = 3d3ae51c29f44826882a9a53da3bc61b
  B = 9a33a847488d4df9b881793fd2bea9af
  C = 1b9eef424e124e899ca2c6324f2473e9
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A   B   C*             |              |              | Yes
 A             | A   B   C*             | 1            | -1           | Yes
 B             | A   B   C*             | 1            | -1           | Yes
 C             | [config not available] |              |              | 

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |        Host         | Replica Count
-------+----------------------------------+---------------------+---------------
 Small | 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:07.688563 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:07.978547 24108 raft_consensus.cc:493] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b9eef424e124e899ca2c6324f2473e9)
I20251024 20:02:07.978663 24108 raft_consensus.cc:515] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } }
I20251024 20:02:07.978896 24108 leader_election.cc:290] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d3ae51c29f44826882a9a53da3bc61b (127.18.153.67:45845), 1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289)
I20251024 20:02:07.979791 24111 raft_consensus.cc:493] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b9eef424e124e899ca2c6324f2473e9)
I20251024 20:02:07.979935 24111 raft_consensus.cc:515] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } }
I20251024 20:02:07.980379 24111 leader_election.cc:290] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a33a847488d4df9b881793fd2bea9af (127.18.153.66:35565), 1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289)
I20251024 20:02:07.982523 23761 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9624e8d395694aa589a3c0e956e8a2d9" candidate_uuid: "9a33a847488d4df9b881793fd2bea9af" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d3ae51c29f44826882a9a53da3bc61b" is_pre_election: true
I20251024 20:02:07.982607 23610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9624e8d395694aa589a3c0e956e8a2d9" candidate_uuid: "9a33a847488d4df9b881793fd2bea9af" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b9eef424e124e899ca2c6324f2473e9" is_pre_election: true
I20251024 20:02:07.982614 23761 raft_consensus.cc:2468] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a33a847488d4df9b881793fd2bea9af in term 1.
W20251024 20:02:07.982841 23652 leader_election.cc:343] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289): Illegal state: cannot vote while shut down
I20251024 20:02:07.982949 23649 leader_election.cc:304] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d3ae51c29f44826882a9a53da3bc61b, 9a33a847488d4df9b881793fd2bea9af; no voters: 1b9eef424e124e899ca2c6324f2473e9
I20251024 20:02:07.983109 24108 raft_consensus.cc:2804] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251024 20:02:07.983146 24108 raft_consensus.cc:493] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b9eef424e124e899ca2c6324f2473e9)
I20251024 20:02:07.983171 24108 raft_consensus.cc:3060] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:02:07.984011 23610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9624e8d395694aa589a3c0e956e8a2d9" candidate_uuid: "3d3ae51c29f44826882a9a53da3bc61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b9eef424e124e899ca2c6324f2473e9" is_pre_election: true
W20251024 20:02:07.984187 23727 leader_election.cc:343] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289): Illegal state: cannot vote while shut down
I20251024 20:02:07.984297 24108 raft_consensus.cc:515] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } }
I20251024 20:02:07.984440 24108 leader_election.cc:290] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [CANDIDATE]: Term 2 election: Requested vote from peers 3d3ae51c29f44826882a9a53da3bc61b (127.18.153.67:45845), 1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289)
I20251024 20:02:07.984508 23686 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9624e8d395694aa589a3c0e956e8a2d9" candidate_uuid: "3d3ae51c29f44826882a9a53da3bc61b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a33a847488d4df9b881793fd2bea9af" is_pre_election: true
I20251024 20:02:07.984551 23610 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9624e8d395694aa589a3c0e956e8a2d9" candidate_uuid: "9a33a847488d4df9b881793fd2bea9af" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b9eef424e124e899ca2c6324f2473e9"
I20251024 20:02:07.984563 23686 raft_consensus.cc:2393] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d3ae51c29f44826882a9a53da3bc61b in current term 2: Already voted for candidate 9a33a847488d4df9b881793fd2bea9af in this term.
W20251024 20:02:07.984714 23652 leader_election.cc:343] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289): Illegal state: cannot vote while shut down
I20251024 20:02:07.984776 23724 leader_election.cc:304] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d3ae51c29f44826882a9a53da3bc61b; no voters: 1b9eef424e124e899ca2c6324f2473e9, 9a33a847488d4df9b881793fd2bea9af
I20251024 20:02:07.984862 24111 raft_consensus.cc:3060] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 1 FOLLOWER]: Advancing to term 2
I20251024 20:02:07.984874 23761 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9624e8d395694aa589a3c0e956e8a2d9" candidate_uuid: "9a33a847488d4df9b881793fd2bea9af" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d3ae51c29f44826882a9a53da3bc61b"
I20251024 20:02:07.985597 24111 raft_consensus.cc:2749] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 20:02:07.986428 23761 raft_consensus.cc:2468] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a33a847488d4df9b881793fd2bea9af in term 2.
I20251024 20:02:07.986605 23649 leader_election.cc:304] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d3ae51c29f44826882a9a53da3bc61b, 9a33a847488d4df9b881793fd2bea9af; no voters: 1b9eef424e124e899ca2c6324f2473e9
I20251024 20:02:07.986699 24108 raft_consensus.cc:2804] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 2 FOLLOWER]: Leader election won for term 2
I20251024 20:02:07.986833 24108 raft_consensus.cc:697] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 2 LEADER]: Becoming Leader. State: Replica: 9a33a847488d4df9b881793fd2bea9af, State: Running, Role: LEADER
I20251024 20:02:07.986883 24108 consensus_queue.cc:237] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } }
I20251024 20:02:07.987483 23504 catalog_manager.cc:5649] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af reported cstate change: term changed from 1 to 2, leader changed from 1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65) to 9a33a847488d4df9b881793fd2bea9af (127.18.153.66). New cstate: current_term: 2 leader_uuid: "9a33a847488d4df9b881793fd2bea9af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b9eef424e124e899ca2c6324f2473e9" member_type: VOTER last_known_addr { host: "127.18.153.65" port: 39289 } health_report { overall_health: UNKNOWN } } }
I20251024 20:02:08.220254 23600 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:08.221177 23751 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:08.221743 23676 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20251024 20:02:08.221834 23826 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 91a71ba4ec254020985ba241455aceef | 127.18.153.126:44001 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1b9eef424e124e899ca2c6324f2473e9 | 127.18.153.65:39289 | HEALTHY | <none>   |       0        |       0
 3d3ae51c29f44826882a9a53da3bc61b | 127.18.153.67:45845 | HEALTHY | <none>   |       0        |       0
 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | HEALTHY | <none>   |       0        |       0
 9a33a847488d4df9b881793fd2bea9af | 127.18.153.66:35565 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
Tablet 9624e8d395694aa589a3c0e956e8a2d9 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  3d3ae51c29f44826882a9a53da3bc61b (127.18.153.67:45845): RUNNING
  9a33a847488d4df9b881793fd2bea9af (127.18.153.66:35565): RUNNING [LEADER]
  1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289): not running
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
  A = 3d3ae51c29f44826882a9a53da3bc61b
  B = 9a33a847488d4df9b881793fd2bea9af
  C = 1b9eef424e124e899ca2c6324f2473e9
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A   B*  C              |              |              | Yes
 A             | A   B   C              | 2            | -1           | Yes
 B             | A   B*  C              | 2            | -1           | Yes
 C             | [config not available] |              |              | 

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |        Host         | Replica Count
-------+----------------------------------+---------------------+---------------
 Small | 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20251024 20:02:08.230785 19045 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20251024 20:02:08.453785 23761 raft_consensus.cc:1275] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 2 FOLLOWER]: Refusing update from remote peer 9a33a847488d4df9b881793fd2bea9af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251024 20:02:08.454056 24108 consensus_queue.cc:1048] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251024 20:02:08.455302 23652 consensus_queue.cc:579] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [LEADER]: The tablet replica hosted on peer 1b9eef424e124e899ca2c6324f2473e9 has failed
W20251024 20:02:08.455406 23652 consensus_peers.cc:597] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af -> Peer 1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289): Couldn't send request to peer 1b9eef424e124e899ca2c6324f2473e9. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20251024 20:02:08.456058 23686 consensus_queue.cc:237] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } }
I20251024 20:02:08.456377 23761 raft_consensus.cc:1275] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 2 FOLLOWER]: Refusing update from remote peer 9a33a847488d4df9b881793fd2bea9af: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251024 20:02:08.456533 24115 consensus_queue.cc:1048] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251024 20:02:08.457041 24108 raft_consensus.cc:2955] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } }
I20251024 20:02:08.457196 23761 raft_consensus.cc:2955] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } }
I20251024 20:02:08.458104 23490 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 9624e8d395694aa589a3c0e956e8a2d9 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251024 20:02:08.458279 23504 catalog_manager.cc:5649] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b reported cstate change: config changed from index -1 to 3, VOTER 1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65) evicted. New cstate: current_term: 2 leader_uuid: "9a33a847488d4df9b881793fd2bea9af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } }
I20251024 20:02:08.459003 23600 tablet_service.cc:1552] Processing DeleteTablet for tablet 9624e8d395694aa589a3c0e956e8a2d9 with delete_type TABLET_DATA_TOMBSTONED (TS 1b9eef424e124e899ca2c6324f2473e9 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58908
I20251024 20:02:08.459160 23686 consensus_queue.cc:237] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "56050fa24271490097d19b69be5da104" member_type: NON_VOTER last_known_addr { host: "127.18.153.68" port: 34307 } attrs { promote: true } }
I20251024 20:02:08.459245 24152 ts_tablet_manager.cc:1916] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251024 20:02:08.459704 23761 raft_consensus.cc:1275] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 2 FOLLOWER]: Refusing update from remote peer 9a33a847488d4df9b881793fd2bea9af: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251024 20:02:08.459894 24151 consensus_queue.cc:1048] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251024 20:02:08.460729 24152 ts_tablet_manager.cc:1929] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20251024 20:02:08.460784 24152 log.cc:1199] T 9624e8d395694aa589a3c0e956e8a2d9 P 1b9eef424e124e899ca2c6324f2473e9: Deleting WAL directory at /tmp/dist-test-tasknDbTLU/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1761336053687453-19045-0/minicluster-data/ts-0-root/wals/9624e8d395694aa589a3c0e956e8a2d9
I20251024 20:02:08.461087 24108 raft_consensus.cc:2955] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER 56050fa24271490097d19b69be5da104 (127.18.153.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "56050fa24271490097d19b69be5da104" member_type: NON_VOTER last_known_addr { host: "127.18.153.68" port: 34307 } attrs { promote: true } } }
I20251024 20:02:08.461162 23493 catalog_manager.cc:4985] TS 1b9eef424e124e899ca2c6324f2473e9 (127.18.153.65:39289): tablet 9624e8d395694aa589a3c0e956e8a2d9 (table test-workload [id=5ef79921f748446f9cae1b44ce4b1df1]) successfully deleted
I20251024 20:02:08.461916 23761 raft_consensus.cc:2955] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER 56050fa24271490097d19b69be5da104 (127.18.153.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "56050fa24271490097d19b69be5da104" member_type: NON_VOTER last_known_addr { host: "127.18.153.68" port: 34307 } attrs { promote: true } } }
I20251024 20:02:08.462327 23490 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9624e8d395694aa589a3c0e956e8a2d9 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20251024 20:02:08.462460 23649 consensus_peers.cc:597] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af -> Peer 56050fa24271490097d19b69be5da104 (127.18.153.68:34307): Couldn't send request to peer 56050fa24271490097d19b69be5da104. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9624e8d395694aa589a3c0e956e8a2d9. This is attempt 1: this message will repeat every 5th retry.
I20251024 20:02:08.462917 23504 catalog_manager.cc:5649] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b reported cstate change: config changed from index 3 to 4, NON_VOTER 56050fa24271490097d19b69be5da104 (127.18.153.68) added. New cstate: current_term: 2 leader_uuid: "9a33a847488d4df9b881793fd2bea9af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d3ae51c29f44826882a9a53da3bc61b" member_type: VOTER last_known_addr { host: "127.18.153.67" port: 45845 } } peers { permanent_uuid: "9a33a847488d4df9b881793fd2bea9af" member_type: VOTER last_known_addr { host: "127.18.153.66" port: 35565 } } peers { permanent_uuid: "56050fa24271490097d19b69be5da104" member_type: NON_VOTER last_known_addr { host: "127.18.153.68" port: 34307 } attrs { promote: true } } }
I20251024 20:02:09.014257 23600 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:09.014362 23751 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:09.015200 23826 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20251024 20:02:09.018106 23676 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 91a71ba4ec254020985ba241455aceef | 127.18.153.126:44001 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1b9eef424e124e899ca2c6324f2473e9 | 127.18.153.65:39289 | HEALTHY | <none>   |       0        |       0
 3d3ae51c29f44826882a9a53da3bc61b | 127.18.153.67:45845 | HEALTHY | <none>   |       0        |       0
 56050fa24271490097d19b69be5da104 | 127.18.153.68:34307 | HEALTHY | <none>   |       0        |       0
 9a33a847488d4df9b881793fd2bea9af | 127.18.153.66:35565 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
     Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |        Host         | Replica Count
-------+----------------------------------+---------------------+---------------
 Small | 1b9eef424e124e899ca2c6324f2473e9 | 127.18.153.65:39289 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20251024 20:02:09.026621 19045 log_verifier.cc:126] Checking tablet 9624e8d395694aa589a3c0e956e8a2d9
I20251024 20:02:09.034274 19045 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 9624e8d395694aa589a3c0e956e8a2d9
I20251024 20:02:09.035071 19045 tablet_server.cc:178] TabletServer@127.18.153.65:0 shutting down...
I20251024 20:02:09.038358 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:09.050521 19045 tablet_server.cc:195] TabletServer@127.18.153.65:0 shutdown complete.
I20251024 20:02:09.051656 19045 tablet_server.cc:178] TabletServer@127.18.153.66:0 shutting down...
I20251024 20:02:09.054502 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:09.054631 19045 tablet_replica.cc:333] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af: stopping tablet replica
I20251024 20:02:09.054724 19045 raft_consensus.cc:2243] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 2 LEADER]: Raft consensus shutting down.
I20251024 20:02:09.054834 19045 raft_consensus.cc:2272] T 9624e8d395694aa589a3c0e956e8a2d9 P 9a33a847488d4df9b881793fd2bea9af [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:09.067792 19045 tablet_server.cc:195] TabletServer@127.18.153.66:0 shutdown complete.
I20251024 20:02:09.068711 19045 tablet_server.cc:178] TabletServer@127.18.153.67:0 shutting down...
I20251024 20:02:09.071798 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:09.071926 19045 tablet_replica.cc:333] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b: stopping tablet replica
I20251024 20:02:09.072017 19045 raft_consensus.cc:2243] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 2 FOLLOWER]: Raft consensus shutting down.
I20251024 20:02:09.072090 19045 raft_consensus.cc:2272] T 9624e8d395694aa589a3c0e956e8a2d9 P 3d3ae51c29f44826882a9a53da3bc61b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:09.084589 19045 tablet_server.cc:195] TabletServer@127.18.153.67:0 shutdown complete.
I20251024 20:02:09.085542 19045 tablet_server.cc:178] TabletServer@127.18.153.68:0 shutting down...
I20251024 20:02:09.088244 19045 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251024 20:02:09.099877 19045 tablet_server.cc:195] TabletServer@127.18.153.68:0 shutdown complete.
I20251024 20:02:09.100680 19045 master.cc:561] Master@127.18.153.126:44001 shutting down...
I20251024 20:02:09.103397 19045 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [term 1 LEADER]: Raft consensus shutting down.
I20251024 20:02:09.103480 19045 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 20:02:09.103502 19045 tablet_replica.cc:333] T 00000000000000000000000000000000 P 91a71ba4ec254020985ba241455aceef: stopping tablet replica
I20251024 20:02:09.115330 19045 master.cc:583] Master@127.18.153.126:44001 shutdown complete.
[       OK ] FailedTabletsAreReplacedITest.OneReplica/1 (3454 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (8025 ms total)

[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (75427 ms total)
[  PASSED  ] 12 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] TsTabletManagerITest.TestTableStats

 1 FAILED TEST
I20251024 20:02:09.120235 19045 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~8 seconds ago
I20251024 20:02:09.120285 19045 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 1 messages since previous log ~8 seconds ago
I20251024 20:02:09.120297 19045 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:542: suppressed but not reported on 1 messages since previous log ~8 seconds ago
I20251024 20:02:09.120321 19045 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5101: suppressed but not reported on 2 messages since previous log ~40 seconds ago
I20251024 20:02:09.120340 19045 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 9 messages since previous log ~42 seconds ago
I20251024 20:02:09.120363 19045 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 437 messages since previous log ~60 seconds ago
I20251024 20:02:09.120380 19045 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tablet/tablet.cc:2378: suppressed but not reported on 7 messages since previous log ~9 seconds ago