TsTabletManagerITest.TestTableStats: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:783: Failure
Expected equality of these values:
  live_row_count
    Which is: 67
  table_info->GetMetrics()->live_row_count->value()
    Which is: 134
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:395: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:767: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
  Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:785: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 782, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
  Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:895: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240507 08:06:39.599926 28050 tablet_server.cc:178] TabletServer@127.27.100.129:0 shutting down...
W20240507 08:06:39.626142 29798 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
W20240507 08:06:39.635788 29798 consensus_peers.cc:485] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a -> Peer 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455): Couldn't send request to peer 3cebffdfceb24f75ae9450fce8382fa5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:39.649138 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:39.649910 28050 tablet_replica.cc:330] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5: stopping tablet replica
I20240507 08:06:39.650496 28050 raft_consensus.cc:2229] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 LEADER]: Raft consensus shutting down.
I20240507 08:06:39.651553 28050 raft_consensus.cc:2258] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.653726 28050 tablet_replica.cc:330] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5: stopping tablet replica
I20240507 08:06:39.654170 28050 raft_consensus.cc:2229] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:39.654618 28050 raft_consensus.cc:2258] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.656384 28050 tablet_replica.cc:330] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5: stopping tablet replica
I20240507 08:06:39.656829 28050 raft_consensus.cc:2229] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:39.657285 28050 raft_consensus.cc:2258] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240507 08:06:39.700012 29798 consensus_peers.cc:485] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a -> Peer 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455): Couldn't send request to peer 3cebffdfceb24f75ae9450fce8382fa5. Status: Network error: Client connection negotiation failed: client connection to 127.27.100.129:35455: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:39.701740 28050 tablet_server.cc:194] TabletServer@127.27.100.129:0 shutdown complete.
I20240507 08:06:39.715780 28050 tablet_server.cc:178] TabletServer@127.27.100.130:0 shutting down...
I20240507 08:06:39.765619 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:39.766323 28050 tablet_replica.cc:330] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a: stopping tablet replica
I20240507 08:06:39.766937 28050 raft_consensus.cc:2229] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:39.767426 28050 raft_consensus.cc:2258] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.769383 28050 tablet_replica.cc:330] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a: stopping tablet replica
I20240507 08:06:39.769915 28050 raft_consensus.cc:2229] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 3 LEADER]: Raft consensus shutting down.
I20240507 08:06:39.770632 28050 raft_consensus.cc:2258] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.772517 28050 tablet_replica.cc:330] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a: stopping tablet replica
I20240507 08:06:39.772948 28050 raft_consensus.cc:2229] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 3 LEADER]: Raft consensus shutting down.
I20240507 08:06:39.773617 28050 raft_consensus.cc:2258] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.784636 30247 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3cebffdfceb24f75ae9450fce8382fa5)
I20240507 08:06:39.785065 30247 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:39.786935 30247 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 3f397b9469554143884dc544bf64282a (127.27.100.130:36791)
W20240507 08:06:39.794309 29876 leader_election.cc:334] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455): Network error: Client connection negotiation failed: client connection to 127.27.100.129:35455: connect: Connection refused (error 111)
I20240507 08:06:39.797348 28050 tablet_server.cc:194] TabletServer@127.27.100.130:0 shutdown complete.
W20240507 08:06:39.798343 29877 leader_election.cc:334] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3f397b9469554143884dc544bf64282a (127.27.100.130:36791): Network error: Client connection negotiation failed: client connection to 127.27.100.130:36791: connect: Connection refused (error 111)
I20240507 08:06:39.798760 29877 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 3cebffdfceb24f75ae9450fce8382fa5, 3f397b9469554143884dc544bf64282a
I20240507 08:06:39.799486 30247 raft_consensus.cc:2726] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:39.811606 28050 tablet_server.cc:178] TabletServer@127.27.100.131:0 shutting down...
I20240507 08:06:39.841075 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:39.841650 28050 tablet_replica.cc:330] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f: stopping tablet replica
I20240507 08:06:39.842520 28050 raft_consensus.cc:2229] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:39.842988 28050 raft_consensus.cc:2258] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.844956 28050 tablet_replica.cc:330] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f: stopping tablet replica
I20240507 08:06:39.845412 28050 raft_consensus.cc:2229] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:39.845881 28050 raft_consensus.cc:2258] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.847631 28050 tablet_replica.cc:330] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f: stopping tablet replica
I20240507 08:06:39.848062 28050 raft_consensus.cc:2229] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:39.848488 28050 raft_consensus.cc:2258] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.870357 28050 tablet_server.cc:194] TabletServer@127.27.100.131:0 shutdown complete.
I20240507 08:06:39.885331 28050 master.cc:537] Master@127.27.100.190:33177 shutting down...
I20240507 08:06:39.910288 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:39.911062 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.911386 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33: stopping tablet replica
I20240507 08:06:39.929682 28050 master.cc:552] Master@127.27.100.190:33177 shutdown complete.
I20240507 08:06:39.947729 28050 master.cc:537] Master@127.27.100.189:36005 shutting down...
W20240507 08:06:39.956987 29538 consensus_peers.cc:485] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 -> Peer 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177): Couldn't send request to peer 8b50ddd4dc5d4bcb84e234822fcc7f33. Status: Network error: Client connection negotiation failed: client connection to 127.27.100.190:33177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:39.969559 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 7 LEADER]: Raft consensus shutting down.
I20240507 08:06:39.970486 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.970820 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5: stopping tablet replica
I20240507 08:06:39.990850 28050 master.cc:552] Master@127.27.100.189:36005 shutdown complete.
I20240507 08:06:40.010131 28050 master.cc:537] Master@127.27.100.188:35355 shutting down...
I20240507 08:06:40.026506 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:40.027032 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:40.027289 28050 tablet_replica.cc:330] stopping tablet replica
I20240507 08:06:40.044183 28050 master.cc:552] Master@127.27.100.188:35355 shutdown complete.
I20240507 08:06:40.069330 28050 test_util.cc:175] -----------------------------------------------
I20240507 08:06:40.069541 28050 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0

[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN      ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20240507 08:05:47.846549 28050 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 1.092s	user 0.427s	sys 0.662s
W20240507 08:05:47.847041 28050 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 1.093s	user 0.427s	sys 0.662s
I20240507 08:05:47.849722 28050 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.100.190:44401
I20240507 08:05:47.851301 28050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 08:05:47.852636 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:47.867017 28061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:47.868503 28062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:47.942700 28050 server_base.cc:1034] running on GCE node
W20240507 08:05:47.943672 28064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:47.944876 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:47.945106 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:47.945262 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069147945248 us; error 0 us; skew 500 ppm
I20240507 08:05:47.945950 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:47.950884 28050 webserver.cc:460] Webserver started at http://127.27.100.190:44435/ using document root <none> and password file <none>
I20240507 08:05:47.951674 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:47.951865 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:47.952230 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:47.957975 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/master-0-root/instance:
uuid: "1f0883b7644d467a873bb847f60869c4"
format_stamp: "Formatted at 2024-05-07 08:05:47 on dist-test-slave-j3cx"
I20240507 08:05:47.965332 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.002s
I20240507 08:05:47.970185 28069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:47.971333 28050 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240507 08:05:47.971694 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/master-0-root
uuid: "1f0883b7644d467a873bb847f60869c4"
format_stamp: "Formatted at 2024-05-07 08:05:47 on dist-test-slave-j3cx"
I20240507 08:05:47.972025 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:47.996155 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:05:47.997545 28050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 08:05:47.997968 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:48.065308 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.190:44401
I20240507 08:05:48.065382 28120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.190:44401 every 8 connection(s)
I20240507 08:05:48.071182 28121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:48.094089 28121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4: Bootstrap starting.
I20240507 08:05:48.099797 28121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:48.101569 28121 log.cc:824] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4: Log is configured to *not* fsync() on all Append() calls
I20240507 08:05:48.105721 28121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4: No bootstrap required, opened a new log
I20240507 08:05:48.127485 28121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0883b7644d467a873bb847f60869c4" member_type: VOTER }
I20240507 08:05:48.128077 28121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:48.128330 28121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0883b7644d467a873bb847f60869c4, State: Initialized, Role: FOLLOWER
I20240507 08:05:48.129055 28121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0883b7644d467a873bb847f60869c4" member_type: VOTER }
I20240507 08:05:48.129520 28121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:05:48.129787 28121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:05:48.130097 28121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:48.133955 28121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0883b7644d467a873bb847f60869c4" member_type: VOTER }
I20240507 08:05:48.134603 28121 leader_election.cc:302] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f0883b7644d467a873bb847f60869c4; no voters: 
I20240507 08:05:48.136236 28121 leader_election.cc:288] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:05:48.136619 28124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:48.139104 28124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0883b7644d467a873bb847f60869c4, State: Running, Role: LEADER
I20240507 08:05:48.139974 28124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0883b7644d467a873bb847f60869c4" member_type: VOTER }
I20240507 08:05:48.140851 28121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:05:48.147136 28125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f0883b7644d467a873bb847f60869c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0883b7644d467a873bb847f60869c4" member_type: VOTER } }
I20240507 08:05:48.147904 28125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [sys.catalog]: This master's current role is: LEADER
I20240507 08:05:48.148123 28126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f0883b7644d467a873bb847f60869c4. Latest consensus state: current_term: 1 leader_uuid: "1f0883b7644d467a873bb847f60869c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0883b7644d467a873bb847f60869c4" member_type: VOTER } }
I20240507 08:05:48.151787 28126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [sys.catalog]: This master's current role is: LEADER
I20240507 08:05:48.155941 28131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:05:48.169236 28131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:05:48.178575 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:05:48.189155 28131 catalog_manager.cc:1347] Generated new cluster ID: 9a324b6083754ec9b3d4c7cccdba11a3
I20240507 08:05:48.189532 28131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:05:48.212819 28131 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:05:48.214313 28131 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:05:48.232930 28131 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4: Generated new TSK 0
I20240507 08:05:48.233817 28131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:05:48.244815 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:48.254280 28143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:48.255642 28144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:48.260737 28146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:48.261567 28050 server_base.cc:1034] running on GCE node
I20240507 08:05:48.262248 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:48.262452 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:48.262629 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069148262612 us; error 0 us; skew 500 ppm
I20240507 08:05:48.263207 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:48.267561 28050 webserver.cc:460] Webserver started at http://127.27.100.129:37073/ using document root <none> and password file <none>
I20240507 08:05:48.268067 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:48.268256 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:48.268533 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:48.269769 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/ts-0-root/instance:
uuid: "014c682f123e4dcabf098c6498433111"
format_stamp: "Formatted at 2024-05-07 08:05:48 on dist-test-slave-j3cx"
I20240507 08:05:48.275534 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240507 08:05:48.279939 28151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:48.280733 28050 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240507 08:05:48.281023 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "014c682f123e4dcabf098c6498433111"
format_stamp: "Formatted at 2024-05-07 08:05:48 on dist-test-slave-j3cx"
I20240507 08:05:48.281302 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:48.313417 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:05:48.314813 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:48.317090 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:05:48.321069 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:05:48.321309 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:48.321558 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:05:48.321722 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:48.417714 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:38373
I20240507 08:05:48.417862 28213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:38373 every 8 connection(s)
I20240507 08:05:48.424815 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:48.435362 28218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:48.437300 28219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:48.445144 28050 server_base.cc:1034] running on GCE node
W20240507 08:05:48.445973 28221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:48.447068 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:48.447332 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:48.447597 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069148447577 us; error 0 us; skew 500 ppm
I20240507 08:05:48.448050 28214 heartbeater.cc:347] Connected to a master server at 127.27.100.190:44401
I20240507 08:05:48.448276 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:48.448513 28214 heartbeater.cc:464] Registering TS with master...
I20240507 08:05:48.449810 28214 heartbeater.cc:511] Master 127.27.100.190:44401 requested a full tablet report, sending...
I20240507 08:05:48.452399 28050 webserver.cc:460] Webserver started at http://127.27.100.130:36751/ using document root <none> and password file <none>
I20240507 08:05:48.452873 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:48.453052 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:48.453305 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:48.453197 28086 ts_manager.cc:194] Registered new tserver with Master: 014c682f123e4dcabf098c6498433111 (127.27.100.129:38373)
I20240507 08:05:48.454507 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/ts-1-root/instance:
uuid: "c3bd37ee69b141b5b2becd803789828e"
format_stamp: "Formatted at 2024-05-07 08:05:48 on dist-test-slave-j3cx"
I20240507 08:05:48.456189 28086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58194
I20240507 08:05:48.460819 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.003s	sys 0.005s
I20240507 08:05:48.465209 28226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:48.465987 28050 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240507 08:05:48.466245 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/ts-1-root
uuid: "c3bd37ee69b141b5b2becd803789828e"
format_stamp: "Formatted at 2024-05-07 08:05:48 on dist-test-slave-j3cx"
I20240507 08:05:48.466511 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1715069146555183-28050-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:48.485932 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:05:48.487144 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:48.489645 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:05:48.493021 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:05:48.493199 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:48.493443 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:05:48.493584 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:48.560103 28214 heartbeater.cc:503] Master 127.27.100.190:44401 was elected leader, sending a full tablet report...
I20240507 08:05:48.588737 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.130:35671
I20240507 08:05:48.588902 28288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.130:35671 every 8 connection(s)
I20240507 08:05:48.605396 28289 heartbeater.cc:347] Connected to a master server at 127.27.100.190:44401
I20240507 08:05:48.605818 28289 heartbeater.cc:464] Registering TS with master...
I20240507 08:05:48.606735 28289 heartbeater.cc:511] Master 127.27.100.190:44401 requested a full tablet report, sending...
I20240507 08:05:48.608808 28086 ts_manager.cc:194] Registered new tserver with Master: c3bd37ee69b141b5b2becd803789828e (127.27.100.130:35671)
I20240507 08:05:48.609167 28050 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.015503936s
I20240507 08:05:48.610538 28086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58206
I20240507 08:05:48.639798 28086 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58218:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 2
    seed: 0
  }
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20240507 08:05:48.661196 28086 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240507 08:05:48.709452 28179 tablet_service.cc:1467] Processing CreateTablet for tablet 23158359aae24c12b7ad69218211e5e8 (DEFAULT_TABLE table=test-table [id=9fde8f08a63d4c988d081352d622b65d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240507 08:05:48.710145 28178 tablet_service.cc:1467] Processing CreateTablet for tablet ed32f70a8e244e36b21c3a3940b7e352 (DEFAULT_TABLE table=test-table [id=9fde8f08a63d4c988d081352d622b65d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240507 08:05:48.711161 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23158359aae24c12b7ad69218211e5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:48.712061 28178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32f70a8e244e36b21c3a3940b7e352. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:48.713311 28289 heartbeater.cc:503] Master 127.27.100.190:44401 was elected leader, sending a full tablet report...
I20240507 08:05:48.720963 28254 tablet_service.cc:1467] Processing CreateTablet for tablet 23158359aae24c12b7ad69218211e5e8 (DEFAULT_TABLE table=test-table [id=9fde8f08a63d4c988d081352d622b65d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240507 08:05:48.721288 28253 tablet_service.cc:1467] Processing CreateTablet for tablet ed32f70a8e244e36b21c3a3940b7e352 (DEFAULT_TABLE table=test-table [id=9fde8f08a63d4c988d081352d622b65d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240507 08:05:48.722345 28254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23158359aae24c12b7ad69218211e5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:48.722967 28253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32f70a8e244e36b21c3a3940b7e352. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:48.745041 28303 tablet_bootstrap.cc:492] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e: Bootstrap starting.
I20240507 08:05:48.752879 28302 tablet_bootstrap.cc:492] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111: Bootstrap starting.
I20240507 08:05:48.753405 28303 tablet_bootstrap.cc:654] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:48.760655 28302 tablet_bootstrap.cc:654] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:48.764020 28303 tablet_bootstrap.cc:492] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e: No bootstrap required, opened a new log
I20240507 08:05:48.764691 28303 ts_tablet_manager.cc:1367] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e: Time spent bootstrapping tablet: real 0.020s	user 0.010s	sys 0.003s
I20240507 08:05:48.767433 28302 tablet_bootstrap.cc:492] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111: No bootstrap required, opened a new log
I20240507 08:05:48.767184 28303 raft_consensus.cc:348] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:48.767939 28302 ts_tablet_manager.cc:1367] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111: Time spent bootstrapping tablet: real 0.015s	user 0.009s	sys 0.004s
I20240507 08:05:48.768085 28303 raft_consensus.cc:729] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3bd37ee69b141b5b2becd803789828e, State: Initialized, Role: FOLLOWER
I20240507 08:05:48.768769 28303 consensus_queue.cc:260] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:48.773841 28303 ts_tablet_manager.cc:1398] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20240507 08:05:48.774740 28303 tablet_bootstrap.cc:492] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e: Bootstrap starting.
I20240507 08:05:48.780603 28302 raft_consensus.cc:348] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:48.781337 28302 raft_consensus.cc:729] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014c682f123e4dcabf098c6498433111, State: Initialized, Role: FOLLOWER
I20240507 08:05:48.782142 28303 tablet_bootstrap.cc:654] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:48.781998 28302 consensus_queue.cc:260] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:48.794104 28303 tablet_bootstrap.cc:492] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e: No bootstrap required, opened a new log
I20240507 08:05:48.794620 28303 ts_tablet_manager.cc:1367] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e: Time spent bootstrapping tablet: real 0.020s	user 0.009s	sys 0.008s
I20240507 08:05:48.798429 28303 raft_consensus.cc:348] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:48.799260 28303 raft_consensus.cc:729] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3bd37ee69b141b5b2becd803789828e, State: Initialized, Role: FOLLOWER
I20240507 08:05:48.799939 28303 consensus_queue.cc:260] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:48.801044 28302 ts_tablet_manager.cc:1398] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111: Time spent starting tablet: real 0.033s	user 0.008s	sys 0.014s
I20240507 08:05:48.802323 28302 tablet_bootstrap.cc:492] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111: Bootstrap starting.
I20240507 08:05:48.805289 28303 ts_tablet_manager.cc:1398] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
I20240507 08:05:48.808247 28302 tablet_bootstrap.cc:654] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:48.822378 28302 tablet_bootstrap.cc:492] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111: No bootstrap required, opened a new log
I20240507 08:05:48.822870 28302 ts_tablet_manager.cc:1367] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111: Time spent bootstrapping tablet: real 0.021s	user 0.007s	sys 0.011s
I20240507 08:05:48.825522 28302 raft_consensus.cc:348] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:48.826339 28302 raft_consensus.cc:729] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014c682f123e4dcabf098c6498433111, State: Initialized, Role: FOLLOWER
I20240507 08:05:48.826917 28302 consensus_queue.cc:260] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:48.829128 28302 ts_tablet_manager.cc:1398] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20240507 08:05:48.841883 28050 ts_tablet_manager-itest.cc:367] Electing peer 2...
I20240507 08:05:48.842191 28050 raft_consensus.cc:412] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [term 0 FOLLOWER]: Emulating election...
I20240507 08:05:48.842424 28050 raft_consensus.cc:3037] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:05:48.844456 28290 tablet.cc:2369] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 08:05:48.846293 28050 raft_consensus.cc:686] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [term 1 LEADER]: Becoming Leader. State: Replica: c3bd37ee69b141b5b2becd803789828e, State: Running, Role: LEADER
I20240507 08:05:48.846894 28050 consensus_queue.cc:237] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:48.850375 28050 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20240507 08:05:48.856353 28084 catalog_manager.cc:5484] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e reported cstate change: term changed from 0 to 1, leader changed from <none> to c3bd37ee69b141b5b2becd803789828e (127.27.100.130). New cstate: current_term: 1 leader_uuid: "c3bd37ee69b141b5b2becd803789828e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:48.876361 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20240507 08:05:48.980295 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20240507 08:05:49.184382 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20240507 08:05:49.483798 28189 raft_consensus.cc:3037] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:49.493953 28313 consensus_queue.cc:1035] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [LEADER]: Connected to new peer: Peer: permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:49.496678 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20240507 08:05:49.900494 28050 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20240507 08:05:49.900802 28050 raft_consensus.cc:412] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 0 FOLLOWER]: Emulating election...
I20240507 08:05:49.901091 28050 raft_consensus.cc:3037] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:49.907193 28050 raft_consensus.cc:686] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 1 LEADER]: Becoming Leader. State: Replica: c3bd37ee69b141b5b2becd803789828e, State: Running, Role: LEADER
I20240507 08:05:49.907919 28050 consensus_queue.cc:237] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:49.910249 28050 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20240507 08:05:49.914346 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20240507 08:05:49.914897 28084 catalog_manager.cc:5484] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e reported cstate change: term changed from 0 to 1, leader changed from <none> to c3bd37ee69b141b5b2becd803789828e (127.27.100.130). New cstate: current_term: 1 leader_uuid: "c3bd37ee69b141b5b2becd803789828e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:50.018155 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20240507 08:05:50.221906 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20240507 08:05:50.508373 28188 raft_consensus.cc:3037] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:50.514003 28313 consensus_queue.cc:1035] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [LEADER]: Connected to new peer: Peer: permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:50.727588 28050 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20240507 08:05:50.727834 28050 raft_consensus.cc:412] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 1 LEADER]: Emulating election...
I20240507 08:05:50.728070 28050 raft_consensus.cc:3032] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:50.728242 28050 raft_consensus.cc:729] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c3bd37ee69b141b5b2becd803789828e, State: Running, Role: LEADER
I20240507 08:05:50.728720 28050 consensus_queue.cc:260] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:50.729501 28050 raft_consensus.cc:3037] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:50.733783 28050 raft_consensus.cc:686] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 2 LEADER]: Becoming Leader. State: Replica: c3bd37ee69b141b5b2becd803789828e, State: Running, Role: LEADER
I20240507 08:05:50.734429 28050 consensus_queue.cc:237] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:50.736011 28050 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20240507 08:05:50.739403 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20240507 08:05:50.739173 28084 catalog_manager.cc:5484] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "c3bd37ee69b141b5b2becd803789828e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:50.842636 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20240507 08:05:51.046036 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20240507 08:05:51.332420 28188 raft_consensus.cc:3037] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:51.337052 28188 raft_consensus.cc:1261] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 2 FOLLOWER]: Refusing update from remote peer c3bd37ee69b141b5b2becd803789828e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:51.338145 28328 consensus_queue.cc:1035] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [LEADER]: Connected to new peer: Peer: permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:51.351567 28050 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20240507 08:05:51.351850 28050 raft_consensus.cc:412] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 2 FOLLOWER]: Emulating election...
I20240507 08:05:51.352070 28050 raft_consensus.cc:3037] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:51.356014 28050 raft_consensus.cc:686] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 3 LEADER]: Becoming Leader. State: Replica: 014c682f123e4dcabf098c6498433111, State: Running, Role: LEADER
I20240507 08:05:51.356549 28050 consensus_queue.cc:237] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:51.357903 28050 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20240507 08:05:51.361373 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20240507 08:05:51.361223 28084 catalog_manager.cc:5484] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 reported cstate change: term changed from 2 to 3, leader changed from c3bd37ee69b141b5b2becd803789828e (127.27.100.130) to 014c682f123e4dcabf098c6498433111 (127.27.100.129). New cstate: current_term: 3 leader_uuid: "014c682f123e4dcabf098c6498433111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:51.465128 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20240507 08:05:51.668264 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20240507 08:05:51.724488 28189 raft_consensus.cc:1226] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 3 LEADER]: Rejecting Update request from peer c3bd37ee69b141b5b2becd803789828e for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:51.725582 28328 consensus_queue.cc:1046] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:51.726294 28328 consensus_queue.cc:1225] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [LEADER]: Peer 014c682f123e4dcabf098c6498433111 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240507 08:05:51.726652 28335 raft_consensus.cc:3032] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:51.726881 28335 raft_consensus.cc:729] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c3bd37ee69b141b5b2becd803789828e, State: Running, Role: LEADER
I20240507 08:05:51.727360 28335 consensus_queue.cc:260] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:51.728060 28335 raft_consensus.cc:3037] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:51.916828 28264 raft_consensus.cc:1261] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 3 FOLLOWER]: Refusing update from remote peer 014c682f123e4dcabf098c6498433111: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:05:51.917876 28341 consensus_queue.cc:1035] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:52.171602 28050 ts_tablet_manager-itest.cc:367] Electing peer 1...
I20240507 08:05:52.171881 28050 raft_consensus.cc:412] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111 [term 1 FOLLOWER]: Emulating election...
I20240507 08:05:52.172123 28050 raft_consensus.cc:3037] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:52.176504 28050 raft_consensus.cc:686] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111 [term 2 LEADER]: Becoming Leader. State: Replica: 014c682f123e4dcabf098c6498433111, State: Running, Role: LEADER
I20240507 08:05:52.177071 28050 consensus_queue.cc:237] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:52.178980 28050 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20240507 08:05:52.182132 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20240507 08:05:52.182330 28085 catalog_manager.cc:5484] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111 reported cstate change: term changed from 1 to 2, leader changed from c3bd37ee69b141b5b2becd803789828e (127.27.100.130) to 014c682f123e4dcabf098c6498433111 (127.27.100.129). New cstate: current_term: 2 leader_uuid: "014c682f123e4dcabf098c6498433111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:52.184944 28189 raft_consensus.cc:1226] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111 [term 2 LEADER]: Rejecting Update request from peer c3bd37ee69b141b5b2becd803789828e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:52.186057 28335 consensus_queue.cc:1046] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:52.186537 28335 consensus_queue.cc:1225] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [LEADER]: Peer 014c682f123e4dcabf098c6498433111 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20240507 08:05:52.186866 28335 raft_consensus.cc:3032] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:52.187100 28335 raft_consensus.cc:729] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c3bd37ee69b141b5b2becd803789828e, State: Running, Role: LEADER
I20240507 08:05:52.187717 28335 consensus_queue.cc:260] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:52.188321 28335 raft_consensus.cc:3037] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:52.285316 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20240507 08:05:52.488735 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20240507 08:05:52.743177 28264 raft_consensus.cc:1261] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [term 2 FOLLOWER]: Refusing update from remote peer 014c682f123e4dcabf098c6498433111: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:52.744324 28341 consensus_queue.cc:1035] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:52.792145 28050 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20240507 08:05:52.792379 28050 raft_consensus.cc:412] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 3 FOLLOWER]: Emulating election...
I20240507 08:05:52.792593 28050 raft_consensus.cc:3037] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:52.796536 28050 raft_consensus.cc:686] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 4 LEADER]: Becoming Leader. State: Replica: c3bd37ee69b141b5b2becd803789828e, State: Running, Role: LEADER
I20240507 08:05:52.797132 28050 consensus_queue.cc:237] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:52.798357 28050 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20240507 08:05:52.801560 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20240507 08:05:52.801908 28084 catalog_manager.cc:5484] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e reported cstate change: term changed from 3 to 4, leader changed from 014c682f123e4dcabf098c6498433111 (127.27.100.129) to c3bd37ee69b141b5b2becd803789828e (127.27.100.130). New cstate: current_term: 4 leader_uuid: "c3bd37ee69b141b5b2becd803789828e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:52.904914 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20240507 08:05:53.055174 28264 raft_consensus.cc:1226] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 4 LEADER]: Rejecting Update request from peer 014c682f123e4dcabf098c6498433111 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:53.056385 28341 consensus_queue.cc:1046] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:53.056933 28341 consensus_queue.cc:1225] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [LEADER]: Peer c3bd37ee69b141b5b2becd803789828e log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240507 08:05:53.057297 28353 raft_consensus.cc:3032] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:53.057551 28353 raft_consensus.cc:729] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 014c682f123e4dcabf098c6498433111, State: Running, Role: LEADER
I20240507 08:05:53.058030 28353 consensus_queue.cc:260] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 } } peers { permanent_uuid: "c3bd37ee69b141b5b2becd803789828e" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 35671 } }
I20240507 08:05:53.058722 28353 raft_consensus.cc:3037] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:53.108260 28050 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20240507 08:05:53.283567 28189 raft_consensus.cc:1261] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 4 FOLLOWER]: Refusing update from remote peer c3bd37ee69b141b5b2becd803789828e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:05:53.284641 28351 consensus_queue.cc:1035] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [LEADER]: Connected to new peer: Peer: permanent_uuid: "014c682f123e4dcabf098c6498433111" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:53.614466 28050 tablet_server.cc:178] TabletServer@127.27.100.129:0 shutting down...
I20240507 08:05:53.632257 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:05:53.632784 28050 tablet_replica.cc:330] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111: stopping tablet replica
I20240507 08:05:53.633337 28050 raft_consensus.cc:2229] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:53.633745 28050 raft_consensus.cc:2258] T 23158359aae24c12b7ad69218211e5e8 P 014c682f123e4dcabf098c6498433111 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:53.635835 28050 tablet_replica.cc:330] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111: stopping tablet replica
I20240507 08:05:53.636230 28050 raft_consensus.cc:2229] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111 [term 2 LEADER]: Raft consensus shutting down.
I20240507 08:05:53.636700 28050 raft_consensus.cc:2258] T ed32f70a8e244e36b21c3a3940b7e352 P 014c682f123e4dcabf098c6498433111 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240507 08:05:53.716820 28230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.100.129:38373: connect: Connection refused (error 111)
I20240507 08:05:53.718483 28050 tablet_server.cc:194] TabletServer@127.27.100.129:0 shutdown complete.
W20240507 08:05:53.720062 28230 consensus_peers.cc:485] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e -> Peer 014c682f123e4dcabf098c6498433111 (127.27.100.129:38373): Couldn't send request to peer 014c682f123e4dcabf098c6498433111. Status: Network error: Client connection negotiation failed: client connection to 127.27.100.129:38373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:53.727480 28050 tablet_server.cc:178] TabletServer@127.27.100.130:0 shutting down...
I20240507 08:05:53.742738 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:05:53.743288 28050 tablet_replica.cc:330] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e: stopping tablet replica
I20240507 08:05:53.743773 28050 raft_consensus.cc:2229] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:53.744159 28050 raft_consensus.cc:2258] T ed32f70a8e244e36b21c3a3940b7e352 P c3bd37ee69b141b5b2becd803789828e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:53.745978 28050 tablet_replica.cc:330] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e: stopping tablet replica
I20240507 08:05:53.746366 28050 raft_consensus.cc:2229] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 4 LEADER]: Raft consensus shutting down.
I20240507 08:05:53.746850 28050 raft_consensus.cc:2258] T 23158359aae24c12b7ad69218211e5e8 P c3bd37ee69b141b5b2becd803789828e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:53.764951 28050 tablet_server.cc:194] TabletServer@127.27.100.130:0 shutdown complete.
I20240507 08:05:53.773484 28050 master.cc:537] Master@127.27.100.190:44401 shutting down...
I20240507 08:05:53.912585 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:05:53.913087 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:53.913398 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1f0883b7644d467a873bb847f60869c4: stopping tablet replica
I20240507 08:05:53.929023 28050 master.cc:552] Master@127.27.100.190:44401 shutdown complete.
[       OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (7243 ms)
[ RUN      ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20240507 08:05:53.956684 28050 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.100.190:42015
I20240507 08:05:53.957609 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:53.963744 28369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:53.965464 28050 server_base.cc:1034] running on GCE node
W20240507 08:05:53.966296 28372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:53.966339 28370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:53.967234 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:53.967420 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:53.967604 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069153967587 us; error 0 us; skew 500 ppm
I20240507 08:05:53.968079 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:53.975199 28050 webserver.cc:460] Webserver started at http://127.27.100.190:45423/ using document root <none> and password file <none>
I20240507 08:05:53.975687 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:53.975867 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:53.976107 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:53.977144 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/master-0-root/instance:
uuid: "274776285c714c609f324ad8e3a6c5cc"
format_stamp: "Formatted at 2024-05-07 08:05:53 on dist-test-slave-j3cx"
I20240507 08:05:53.981159 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.002s
I20240507 08:05:53.984140 28377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:53.984782 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240507 08:05:53.985056 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/master-0-root
uuid: "274776285c714c609f324ad8e3a6c5cc"
format_stamp: "Formatted at 2024-05-07 08:05:53 on dist-test-slave-j3cx"
I20240507 08:05:53.985304 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:54.008071 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:05:54.009044 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:54.041893 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.190:42015
I20240507 08:05:54.041972 28428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.190:42015 every 8 connection(s)
I20240507 08:05:54.045524 28429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:54.054073 28429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc: Bootstrap starting.
I20240507 08:05:54.057709 28429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:54.061193 28429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc: No bootstrap required, opened a new log
I20240507 08:05:54.062866 28429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274776285c714c609f324ad8e3a6c5cc" member_type: VOTER }
I20240507 08:05:54.063226 28429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:54.063401 28429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274776285c714c609f324ad8e3a6c5cc, State: Initialized, Role: FOLLOWER
I20240507 08:05:54.063880 28429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274776285c714c609f324ad8e3a6c5cc" member_type: VOTER }
I20240507 08:05:54.064281 28429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:05:54.064465 28429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:05:54.064663 28429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:54.068275 28429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274776285c714c609f324ad8e3a6c5cc" member_type: VOTER }
I20240507 08:05:54.068706 28429 leader_election.cc:302] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 274776285c714c609f324ad8e3a6c5cc; no voters: 
I20240507 08:05:54.069685 28429 leader_election.cc:288] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:05:54.070067 28432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:54.071313 28432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [term 1 LEADER]: Becoming Leader. State: Replica: 274776285c714c609f324ad8e3a6c5cc, State: Running, Role: LEADER
I20240507 08:05:54.071864 28432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274776285c714c609f324ad8e3a6c5cc" member_type: VOTER }
I20240507 08:05:54.072353 28429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:05:54.074759 28434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 274776285c714c609f324ad8e3a6c5cc. Latest consensus state: current_term: 1 leader_uuid: "274776285c714c609f324ad8e3a6c5cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274776285c714c609f324ad8e3a6c5cc" member_type: VOTER } }
I20240507 08:05:54.074715 28433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "274776285c714c609f324ad8e3a6c5cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274776285c714c609f324ad8e3a6c5cc" member_type: VOTER } }
I20240507 08:05:54.076100 28434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [sys.catalog]: This master's current role is: LEADER
I20240507 08:05:54.076166 28433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [sys.catalog]: This master's current role is: LEADER
I20240507 08:05:54.078101 28439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:05:54.082446 28439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:05:54.086305 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:05:54.090057 28439 catalog_manager.cc:1347] Generated new cluster ID: d3c620fc48df4f34832183a4b73620de
I20240507 08:05:54.090322 28439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:05:54.111029 28439 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:05:54.112260 28439 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:05:54.125710 28439 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc: Generated new TSK 0
I20240507 08:05:54.126282 28439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:05:54.152449 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:54.158989 28450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:54.160001 28451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:54.161531 28453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:54.162426 28050 server_base.cc:1034] running on GCE node
I20240507 08:05:54.163103 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:54.163286 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:54.163437 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069154163421 us; error 0 us; skew 500 ppm
I20240507 08:05:54.163987 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:54.166229 28050 webserver.cc:460] Webserver started at http://127.27.100.129:43799/ using document root <none> and password file <none>
I20240507 08:05:54.166661 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:54.166818 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:54.167048 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:54.168113 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/ts-0-root/instance:
uuid: "12bbb534c6fa4f34ab43f3be6f073d03"
format_stamp: "Formatted at 2024-05-07 08:05:54 on dist-test-slave-j3cx"
I20240507 08:05:54.172122 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.002s
I20240507 08:05:54.175037 28458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:54.175762 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240507 08:05:54.176007 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "12bbb534c6fa4f34ab43f3be6f073d03"
format_stamp: "Formatted at 2024-05-07 08:05:54 on dist-test-slave-j3cx"
I20240507 08:05:54.176262 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:54.199347 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:05:54.200349 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:54.201638 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:05:54.203668 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:05:54.203836 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:54.204062 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:05:54.204205 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:54.238652 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:45705
I20240507 08:05:54.238747 28520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:45705 every 8 connection(s)
I20240507 08:05:54.243055 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:54.251385 28525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:54.252097 28526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:54.254678 28521 heartbeater.cc:347] Connected to a master server at 127.27.100.190:42015
I20240507 08:05:54.255030 28521 heartbeater.cc:464] Registering TS with master...
I20240507 08:05:54.255081 28050 server_base.cc:1034] running on GCE node
I20240507 08:05:54.255889 28521 heartbeater.cc:511] Master 127.27.100.190:42015 requested a full tablet report, sending...
W20240507 08:05:54.256860 28528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:54.257665 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:54.257848 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:54.257879 28394 ts_manager.cc:194] Registered new tserver with Master: 12bbb534c6fa4f34ab43f3be6f073d03 (127.27.100.129:45705)
I20240507 08:05:54.258018 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069154258004 us; error 0 us; skew 500 ppm
I20240507 08:05:54.258751 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:54.259860 28394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50566
I20240507 08:05:54.261704 28050 webserver.cc:460] Webserver started at http://127.27.100.130:46107/ using document root <none> and password file <none>
I20240507 08:05:54.262250 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:54.262414 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:54.262660 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:54.263690 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/ts-1-root/instance:
uuid: "5918fa9003ba4ba5ac222e537fab7226"
format_stamp: "Formatted at 2024-05-07 08:05:54 on dist-test-slave-j3cx"
I20240507 08:05:54.267649 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.002s
I20240507 08:05:54.270576 28533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:54.271302 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240507 08:05:54.271567 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/ts-1-root
uuid: "5918fa9003ba4ba5ac222e537fab7226"
format_stamp: "Formatted at 2024-05-07 08:05:54 on dist-test-slave-j3cx"
I20240507 08:05:54.271814 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1715069146555183-28050-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:54.282505 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:05:54.283422 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:54.284693 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:05:54.286818 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:05:54.287016 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:54.287220 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:05:54.287345 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:54.321533 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.130:32981
I20240507 08:05:54.321604 28595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.130:32981 every 8 connection(s)
I20240507 08:05:54.334476 28596 heartbeater.cc:347] Connected to a master server at 127.27.100.190:42015
I20240507 08:05:54.334790 28596 heartbeater.cc:464] Registering TS with master...
I20240507 08:05:54.335587 28596 heartbeater.cc:511] Master 127.27.100.190:42015 requested a full tablet report, sending...
I20240507 08:05:54.337265 28394 ts_manager.cc:194] Registered new tserver with Master: 5918fa9003ba4ba5ac222e537fab7226 (127.27.100.130:32981)
I20240507 08:05:54.337345 28050 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.012937238s
I20240507 08:05:54.338496 28394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50582
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:421: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20240507 08:05:54.356547 28050 tablet_server.cc:178] TabletServer@127.27.100.129:0 shutting down...
I20240507 08:05:54.362339 28521 heartbeater.cc:503] Master 127.27.100.190:42015 was elected leader, sending a full tablet report...
I20240507 08:05:54.371670 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:05:54.386644 28050 tablet_server.cc:194] TabletServer@127.27.100.129:0 shutdown complete.
I20240507 08:05:54.392710 28050 tablet_server.cc:178] TabletServer@127.27.100.130:0 shutting down...
I20240507 08:05:54.405230 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:05:54.420092 28050 tablet_server.cc:194] TabletServer@127.27.100.130:0 shutdown complete.
I20240507 08:05:54.426098 28050 master.cc:537] Master@127.27.100.190:42015 shutting down...
I20240507 08:05:54.436959 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:05:54.437422 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:54.437798 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P 274776285c714c609f324ad8e3a6c5cc: stopping tablet replica
I20240507 08:05:54.455137 28050 master.cc:552] Master@127.27.100.190:42015 shutdown complete.
[  SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (519 ms)
[ RUN      ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20240507 08:05:54.475929 28050 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.100.190:41483
I20240507 08:05:54.476847 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:54.482602 28609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:54.483402 28610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:54.484448 28612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:54.485363 28050 server_base.cc:1034] running on GCE node
I20240507 08:05:54.486006 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:54.486193 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:54.486335 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069154486319 us; error 0 us; skew 500 ppm
I20240507 08:05:54.486785 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:54.488993 28050 webserver.cc:460] Webserver started at http://127.27.100.190:37697/ using document root <none> and password file <none>
I20240507 08:05:54.489425 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:54.489600 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:54.489833 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:54.492588 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/master-0-root/instance:
uuid: "15d065a824de4600b053f1a54f20e418"
format_stamp: "Formatted at 2024-05-07 08:05:54 on dist-test-slave-j3cx"
I20240507 08:05:54.496659 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240507 08:05:54.499820 28617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:54.500527 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240507 08:05:54.500751 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/master-0-root
uuid: "15d065a824de4600b053f1a54f20e418"
format_stamp: "Formatted at 2024-05-07 08:05:54 on dist-test-slave-j3cx"
I20240507 08:05:54.500993 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:54.517621 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:05:54.518546 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:54.550933 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.190:41483
I20240507 08:05:54.551035 28668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.190:41483 every 8 connection(s)
I20240507 08:05:54.554535 28669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:54.563159 28669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418: Bootstrap starting.
I20240507 08:05:54.566699 28669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:54.570299 28669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418: No bootstrap required, opened a new log
I20240507 08:05:54.572165 28669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d065a824de4600b053f1a54f20e418" member_type: VOTER }
I20240507 08:05:54.572609 28669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:54.572845 28669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d065a824de4600b053f1a54f20e418, State: Initialized, Role: FOLLOWER
I20240507 08:05:54.573410 28669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d065a824de4600b053f1a54f20e418" member_type: VOTER }
I20240507 08:05:54.573912 28669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:05:54.574138 28669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:05:54.574385 28669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:54.578522 28669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d065a824de4600b053f1a54f20e418" member_type: VOTER }
I20240507 08:05:54.579039 28669 leader_election.cc:302] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15d065a824de4600b053f1a54f20e418; no voters: 
I20240507 08:05:54.580135 28669 leader_election.cc:288] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:05:54.580385 28672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:54.581712 28672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [term 1 LEADER]: Becoming Leader. State: Replica: 15d065a824de4600b053f1a54f20e418, State: Running, Role: LEADER
I20240507 08:05:54.582324 28672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d065a824de4600b053f1a54f20e418" member_type: VOTER }
I20240507 08:05:54.582950 28669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:05:54.584961 28673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15d065a824de4600b053f1a54f20e418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d065a824de4600b053f1a54f20e418" member_type: VOTER } }
I20240507 08:05:54.585001 28674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15d065a824de4600b053f1a54f20e418. Latest consensus state: current_term: 1 leader_uuid: "15d065a824de4600b053f1a54f20e418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d065a824de4600b053f1a54f20e418" member_type: VOTER } }
I20240507 08:05:54.585724 28674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [sys.catalog]: This master's current role is: LEADER
I20240507 08:05:54.586822 28673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [sys.catalog]: This master's current role is: LEADER
I20240507 08:05:54.588388 28679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:05:54.592994 28679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:05:54.596767 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:05:54.600629 28679 catalog_manager.cc:1347] Generated new cluster ID: 322883853fdc45de8499cbe0b735e2e5
I20240507 08:05:54.600914 28679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:05:54.625097 28679 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:05:54.626274 28679 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:05:54.641362 28679 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418: Generated new TSK 0
I20240507 08:05:54.641953 28679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:05:54.662886 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:54.669505 28690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:54.670362 28691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:54.671382 28693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:54.672305 28050 server_base.cc:1034] running on GCE node
I20240507 08:05:54.672995 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:54.673209 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:54.673360 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069154673347 us; error 0 us; skew 500 ppm
I20240507 08:05:54.673873 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:54.676270 28050 webserver.cc:460] Webserver started at http://127.27.100.129:35481/ using document root <none> and password file <none>
I20240507 08:05:54.676692 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:54.676862 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:54.677095 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:54.678085 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/ts-0-root/instance:
uuid: "ab14b469d965408386b2861733c08dec"
format_stamp: "Formatted at 2024-05-07 08:05:54 on dist-test-slave-j3cx"
I20240507 08:05:54.682126 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240507 08:05:54.685170 28698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:54.685818 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240507 08:05:54.686069 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "ab14b469d965408386b2861733c08dec"
format_stamp: "Formatted at 2024-05-07 08:05:54 on dist-test-slave-j3cx"
I20240507 08:05:54.686323 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:54.695734 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:05:54.696681 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:54.697932 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:05:54.699957 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:05:54.700129 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:54.700353 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:05:54.700488 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:54.735037 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:34941
I20240507 08:05:54.735100 28760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:34941 every 8 connection(s)
I20240507 08:05:54.739226 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:54.747047 28765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:54.748039 28766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:54.749845 28768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:54.749962 28761 heartbeater.cc:347] Connected to a master server at 127.27.100.190:41483
I20240507 08:05:54.750350 28761 heartbeater.cc:464] Registering TS with master...
I20240507 08:05:54.750808 28050 server_base.cc:1034] running on GCE node
I20240507 08:05:54.750984 28761 heartbeater.cc:511] Master 127.27.100.190:41483 requested a full tablet report, sending...
I20240507 08:05:54.751581 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:54.751765 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:54.751906 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069154751890 us; error 0 us; skew 500 ppm
I20240507 08:05:54.752570 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:54.752825 28634 ts_manager.cc:194] Registered new tserver with Master: ab14b469d965408386b2861733c08dec (127.27.100.129:34941)
I20240507 08:05:54.755064 28634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33340
I20240507 08:05:54.755178 28050 webserver.cc:460] Webserver started at http://127.27.100.130:40299/ using document root <none> and password file <none>
I20240507 08:05:54.755785 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:54.756024 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:54.756263 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:54.757580 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/ts-1-root/instance:
uuid: "14d71e3011434aa395a93f2de05f81ec"
format_stamp: "Formatted at 2024-05-07 08:05:54 on dist-test-slave-j3cx"
I20240507 08:05:54.761564 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240507 08:05:54.764581 28773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:54.765372 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240507 08:05:54.765619 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/ts-1-root
uuid: "14d71e3011434aa395a93f2de05f81ec"
format_stamp: "Formatted at 2024-05-07 08:05:54 on dist-test-slave-j3cx"
I20240507 08:05:54.765867 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1715069146555183-28050-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:54.794735 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:05:54.795759 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:54.797056 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:05:54.799196 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:05:54.799387 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:54.799623 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:05:54.799772 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:54.834270 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.130:44947
I20240507 08:05:54.834338 28835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.130:44947 every 8 connection(s)
I20240507 08:05:54.845608 28836 heartbeater.cc:347] Connected to a master server at 127.27.100.190:41483
I20240507 08:05:54.845917 28836 heartbeater.cc:464] Registering TS with master...
I20240507 08:05:54.846525 28836 heartbeater.cc:511] Master 127.27.100.190:41483 requested a full tablet report, sending...
I20240507 08:05:54.848145 28634 ts_manager.cc:194] Registered new tserver with Master: 14d71e3011434aa395a93f2de05f81ec (127.27.100.130:44947)
I20240507 08:05:54.848873 28050 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.011877587s
I20240507 08:05:54.849653 28634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33350
I20240507 08:05:54.857614 28761 heartbeater.cc:503] Master 127.27.100.190:41483 was elected leader, sending a full tablet report...
I20240507 08:05:54.868832 28634 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33362:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20240507 08:05:54.870048 28634 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240507 08:05:54.897715 28801 tablet_service.cc:1467] Processing CreateTablet for tablet df5d26b394024120aab5d665cde96ca9 (DEFAULT_TABLE table=test-table [id=ee02b907967643ac8164fade8b40b584]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:05:54.898845 28801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5d26b394024120aab5d665cde96ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:54.898917 28726 tablet_service.cc:1467] Processing CreateTablet for tablet df5d26b394024120aab5d665cde96ca9 (DEFAULT_TABLE table=test-table [id=ee02b907967643ac8164fade8b40b584]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:05:54.900110 28726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5d26b394024120aab5d665cde96ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:54.914240 28849 tablet_bootstrap.cc:492] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec: Bootstrap starting.
I20240507 08:05:54.915428 28850 tablet_bootstrap.cc:492] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec: Bootstrap starting.
I20240507 08:05:54.918722 28849 tablet_bootstrap.cc:654] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:54.919360 28850 tablet_bootstrap.cc:654] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:54.923058 28849 tablet_bootstrap.cc:492] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec: No bootstrap required, opened a new log
I20240507 08:05:54.923413 28850 tablet_bootstrap.cc:492] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec: No bootstrap required, opened a new log
I20240507 08:05:54.923440 28849 ts_tablet_manager.cc:1367] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec: Time spent bootstrapping tablet: real 0.010s	user 0.007s	sys 0.000s
I20240507 08:05:54.923863 28850 ts_tablet_manager.cc:1367] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.004s
I20240507 08:05:54.925334 28849 raft_consensus.cc:348] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d71e3011434aa395a93f2de05f81ec" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44947 } } peers { permanent_uuid: "ab14b469d965408386b2861733c08dec" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 34941 } }
I20240507 08:05:54.925455 28850 raft_consensus.cc:348] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d71e3011434aa395a93f2de05f81ec" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44947 } } peers { permanent_uuid: "ab14b469d965408386b2861733c08dec" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 34941 } }
I20240507 08:05:54.925814 28849 raft_consensus.cc:729] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d71e3011434aa395a93f2de05f81ec, State: Initialized, Role: FOLLOWER
I20240507 08:05:54.925949 28850 raft_consensus.cc:729] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab14b469d965408386b2861733c08dec, State: Initialized, Role: FOLLOWER
I20240507 08:05:54.926307 28849 consensus_queue.cc:260] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d71e3011434aa395a93f2de05f81ec" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44947 } } peers { permanent_uuid: "ab14b469d965408386b2861733c08dec" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 34941 } }
I20240507 08:05:54.926415 28850 consensus_queue.cc:260] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d71e3011434aa395a93f2de05f81ec" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44947 } } peers { permanent_uuid: "ab14b469d965408386b2861733c08dec" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 34941 } }
I20240507 08:05:54.928995 28836 heartbeater.cc:503] Master 127.27.100.190:41483 was elected leader, sending a full tablet report...
I20240507 08:05:54.929510 28849 ts_tablet_manager.cc:1398] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240507 08:05:54.929833 28850 ts_tablet_manager.cc:1398] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240507 08:05:54.958679 28050 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20240507 08:05:54.958887 28050 raft_consensus.cc:412] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 0 FOLLOWER]: Emulating election...
I20240507 08:05:54.959106 28050 raft_consensus.cc:3037] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:54.963582 28050 raft_consensus.cc:686] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 1 LEADER]: Becoming Leader. State: Replica: ab14b469d965408386b2861733c08dec, State: Running, Role: LEADER
W20240507 08:05:54.963902 28761 heartbeater.cc:644] Failed to heartbeat to 127.27.100.190:41483 (0 consecutive failures): IO error: failing all heartbeats for tests
I20240507 08:05:54.964169 28050 consensus_queue.cc:237] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d71e3011434aa395a93f2de05f81ec" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44947 } } peers { permanent_uuid: "ab14b469d965408386b2861733c08dec" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 34941 } }
I20240507 08:05:54.966861 28050 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20240507 08:05:54.982944 28050 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20240507 08:05:55.087934 28050 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20240507 08:05:55.291654 28050 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20240507 08:05:55.467851 28811 raft_consensus.cc:3037] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:55.472941 28854 consensus_queue.cc:1035] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d71e3011434aa395a93f2de05f81ec" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:55.596236 28050 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20240507 08:05:55.596457 28050 raft_consensus.cc:412] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 1 LEADER]: Emulating election...
I20240507 08:05:55.596637 28050 raft_consensus.cc:3032] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:55.596786 28050 raft_consensus.cc:729] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab14b469d965408386b2861733c08dec, State: Running, Role: LEADER
I20240507 08:05:55.597167 28050 consensus_queue.cc:260] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d71e3011434aa395a93f2de05f81ec" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44947 } } peers { permanent_uuid: "ab14b469d965408386b2861733c08dec" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 34941 } }
I20240507 08:05:55.597760 28050 raft_consensus.cc:3037] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:55.601559 28050 raft_consensus.cc:686] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 2 LEADER]: Becoming Leader. State: Replica: ab14b469d965408386b2861733c08dec, State: Running, Role: LEADER
I20240507 08:05:55.602025 28050 consensus_queue.cc:237] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d71e3011434aa395a93f2de05f81ec" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44947 } } peers { permanent_uuid: "ab14b469d965408386b2861733c08dec" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 34941 } }
I20240507 08:05:55.603258 28050 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20240507 08:05:55.606595 28050 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20240507 08:05:55.710460 28050 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20240507 08:05:55.914160 28050 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20240507 08:05:56.016790 28811 raft_consensus.cc:3037] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:56.021083 28811 raft_consensus.cc:1261] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec [term 2 FOLLOWER]: Refusing update from remote peer ab14b469d965408386b2861733c08dec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:56.022167 28854 consensus_queue.cc:1035] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d71e3011434aa395a93f2de05f81ec" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:56.218932 28050 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20240507 08:05:56.219157 28050 raft_consensus.cc:412] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 2 LEADER]: Emulating election...
I20240507 08:05:56.219326 28050 raft_consensus.cc:3032] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:56.219472 28050 raft_consensus.cc:729] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab14b469d965408386b2861733c08dec, State: Running, Role: LEADER
I20240507 08:05:56.219885 28050 consensus_queue.cc:260] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d71e3011434aa395a93f2de05f81ec" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44947 } } peers { permanent_uuid: "ab14b469d965408386b2861733c08dec" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 34941 } }
I20240507 08:05:56.220472 28050 raft_consensus.cc:3037] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:56.224259 28050 raft_consensus.cc:686] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 3 LEADER]: Becoming Leader. State: Replica: ab14b469d965408386b2861733c08dec, State: Running, Role: LEADER
I20240507 08:05:56.224752 28050 consensus_queue.cc:237] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d71e3011434aa395a93f2de05f81ec" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44947 } } peers { permanent_uuid: "ab14b469d965408386b2861733c08dec" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 34941 } }
I20240507 08:05:56.225999 28050 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20240507 08:05:56.229038 28050 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20240507 08:05:56.332798 28050 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20240507 08:05:56.536445 28050 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20240507 08:05:56.798357 28811 raft_consensus.cc:3037] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:56.804371 28811 raft_consensus.cc:1261] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec [term 3 FOLLOWER]: Refusing update from remote peer ab14b469d965408386b2861733c08dec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:05:56.805799 28868 consensus_queue.cc:1035] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d71e3011434aa395a93f2de05f81ec" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:56.846971 28050 tablet_server.cc:178] TabletServer@127.27.100.129:0 shutting down...
I20240507 08:05:56.867883 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:05:56.868526 28050 tablet_replica.cc:330] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec: stopping tablet replica
I20240507 08:05:56.869061 28050 raft_consensus.cc:2229] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 3 LEADER]: Raft consensus shutting down.
I20240507 08:05:56.869614 28050 raft_consensus.cc:2258] T df5d26b394024120aab5d665cde96ca9 P ab14b469d965408386b2861733c08dec [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:56.889127 28050 tablet_server.cc:194] TabletServer@127.27.100.129:0 shutdown complete.
I20240507 08:05:56.898094 28050 tablet_server.cc:178] TabletServer@127.27.100.130:0 shutting down...
I20240507 08:05:56.916244 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:05:56.916823 28050 tablet_replica.cc:330] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec: stopping tablet replica
I20240507 08:05:56.917273 28050 raft_consensus.cc:2229] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:56.917661 28050 raft_consensus.cc:2258] T df5d26b394024120aab5d665cde96ca9 P 14d71e3011434aa395a93f2de05f81ec [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:56.935688 28050 tablet_server.cc:194] TabletServer@127.27.100.130:0 shutdown complete.
I20240507 08:05:56.943609 28050 master.cc:537] Master@127.27.100.190:41483 shutting down...
I20240507 08:05:56.957662 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:05:56.958264 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:56.958559 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P 15d065a824de4600b053f1a54f20e418: stopping tablet replica
I20240507 08:05:56.975359 28050 master.cc:552] Master@127.27.100.190:41483 shutdown complete.
[       OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (2528 ms)
[----------] 3 tests from LeadershipChangeReportingTest (10292 ms total)

[----------] 4 tests from TsTabletManagerITest
[ RUN      ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20240507 08:05:57.005066 28050 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.100.190:35397
I20240507 08:05:57.006117 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:57.012537 28877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:57.013312 28878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:57.014150 28880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:57.015089 28050 server_base.cc:1034] running on GCE node
I20240507 08:05:57.015781 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:57.015966 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:57.016105 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069157016089 us; error 0 us; skew 500 ppm
I20240507 08:05:57.016633 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:57.019061 28050 webserver.cc:460] Webserver started at http://127.27.100.190:44299/ using document root <none> and password file <none>
I20240507 08:05:57.019570 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:57.019752 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:57.020022 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:57.021163 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/master-0-root/instance:
uuid: "2b9b8fdba1934aba8c13894aba1998fd"
format_stamp: "Formatted at 2024-05-07 08:05:57 on dist-test-slave-j3cx"
I20240507 08:05:57.025480 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240507 08:05:57.028710 28885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:57.029456 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240507 08:05:57.029700 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/master-0-root
uuid: "2b9b8fdba1934aba8c13894aba1998fd"
format_stamp: "Formatted at 2024-05-07 08:05:57 on dist-test-slave-j3cx"
I20240507 08:05:57.029934 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:57.044256 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:05:57.045440 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:57.081450 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.190:35397
I20240507 08:05:57.081542 28936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.190:35397 every 8 connection(s)
I20240507 08:05:57.085155 28937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:57.094429 28937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd: Bootstrap starting.
I20240507 08:05:57.098419 28937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:57.102339 28937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd: No bootstrap required, opened a new log
I20240507 08:05:57.104215 28937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9b8fdba1934aba8c13894aba1998fd" member_type: VOTER }
I20240507 08:05:57.104606 28937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:57.104822 28937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9b8fdba1934aba8c13894aba1998fd, State: Initialized, Role: FOLLOWER
I20240507 08:05:57.105356 28937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9b8fdba1934aba8c13894aba1998fd" member_type: VOTER }
I20240507 08:05:57.105773 28937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:05:57.105980 28937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:05:57.106226 28937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:57.109997 28937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9b8fdba1934aba8c13894aba1998fd" member_type: VOTER }
I20240507 08:05:57.110466 28937 leader_election.cc:302] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b9b8fdba1934aba8c13894aba1998fd; no voters: 
I20240507 08:05:57.111529 28937 leader_election.cc:288] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:05:57.111894 28940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:57.113152 28940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [term 1 LEADER]: Becoming Leader. State: Replica: 2b9b8fdba1934aba8c13894aba1998fd, State: Running, Role: LEADER
I20240507 08:05:57.113703 28940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9b8fdba1934aba8c13894aba1998fd" member_type: VOTER }
I20240507 08:05:57.114249 28937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:05:57.116494 28942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b9b8fdba1934aba8c13894aba1998fd. Latest consensus state: current_term: 1 leader_uuid: "2b9b8fdba1934aba8c13894aba1998fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9b8fdba1934aba8c13894aba1998fd" member_type: VOTER } }
I20240507 08:05:57.116494 28941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b9b8fdba1934aba8c13894aba1998fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9b8fdba1934aba8c13894aba1998fd" member_type: VOTER } }
I20240507 08:05:57.117328 28942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [sys.catalog]: This master's current role is: LEADER
I20240507 08:05:57.117403 28941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [sys.catalog]: This master's current role is: LEADER
I20240507 08:05:57.120957 28946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:05:57.126652 28946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:05:57.128305 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:05:57.134910 28946 catalog_manager.cc:1347] Generated new cluster ID: b6099feca35e429daed083b0d05fc1e1
I20240507 08:05:57.135164 28946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:05:57.168869 28946 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:05:57.170116 28946 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:05:57.186419 28946 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd: Generated new TSK 0
I20240507 08:05:57.186945 28946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:05:57.193980 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:57.200461 28958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:57.201356 28959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:57.202365 28961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:57.203243 28050 server_base.cc:1034] running on GCE node
I20240507 08:05:57.203987 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:57.204154 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:57.204277 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069157204267 us; error 0 us; skew 500 ppm
I20240507 08:05:57.204769 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:57.206990 28050 webserver.cc:460] Webserver started at http://127.27.100.129:44549/ using document root <none> and password file <none>
I20240507 08:05:57.207433 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:57.207644 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:57.207871 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:57.208925 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-0-root/instance:
uuid: "b0cddd4e15cb42e886e4671143b41ff3"
format_stamp: "Formatted at 2024-05-07 08:05:57 on dist-test-slave-j3cx"
I20240507 08:05:57.213174 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240507 08:05:57.216406 28966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:57.217144 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240507 08:05:57.217375 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "b0cddd4e15cb42e886e4671143b41ff3"
format_stamp: "Formatted at 2024-05-07 08:05:57 on dist-test-slave-j3cx"
I20240507 08:05:57.217624 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:57.227912 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:05:57.228899 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:57.230198 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:05:57.232514 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:05:57.232690 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:57.232869 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:05:57.233004 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:57.270166 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:36845
I20240507 08:05:57.270246 29028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:36845 every 8 connection(s)
I20240507 08:05:57.274534 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:57.283444 29033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:57.284448 29034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:57.286995 29036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:57.287333 29029 heartbeater.cc:347] Connected to a master server at 127.27.100.190:35397
I20240507 08:05:57.287779 29029 heartbeater.cc:464] Registering TS with master...
I20240507 08:05:57.288028 28050 server_base.cc:1034] running on GCE node
I20240507 08:05:57.288617 29029 heartbeater.cc:511] Master 127.27.100.190:35397 requested a full tablet report, sending...
I20240507 08:05:57.288722 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:57.288967 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:57.289120 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069157289098 us; error 0 us; skew 500 ppm
I20240507 08:05:57.289813 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:57.290462 28902 ts_manager.cc:194] Registered new tserver with Master: b0cddd4e15cb42e886e4671143b41ff3 (127.27.100.129:36845)
I20240507 08:05:57.292362 28050 webserver.cc:460] Webserver started at http://127.27.100.130:32975/ using document root <none> and password file <none>
I20240507 08:05:57.292418 28902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59558
I20240507 08:05:57.292896 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:57.293092 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:57.293408 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:57.294543 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-1-root/instance:
uuid: "cdca223a8c254da8b83a7786a86ae2be"
format_stamp: "Formatted at 2024-05-07 08:05:57 on dist-test-slave-j3cx"
I20240507 08:05:57.298844 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240507 08:05:57.302080 29041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:57.302752 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240507 08:05:57.303014 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-1-root
uuid: "cdca223a8c254da8b83a7786a86ae2be"
format_stamp: "Formatted at 2024-05-07 08:05:57 on dist-test-slave-j3cx"
I20240507 08:05:57.303279 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:57.322568 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:05:57.323869 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:57.325176 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:05:57.327178 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:05:57.327375 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:57.327601 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:05:57.327751 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:57.365713 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.130:41547
I20240507 08:05:57.365792 29103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.130:41547 every 8 connection(s)
I20240507 08:05:57.370175 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:57.377861 29107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:57.379709 29108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:57.380210 29104 heartbeater.cc:347] Connected to a master server at 127.27.100.190:35397
I20240507 08:05:57.380528 29104 heartbeater.cc:464] Registering TS with master...
I20240507 08:05:57.381011 28050 server_base.cc:1034] running on GCE node
I20240507 08:05:57.381378 29104 heartbeater.cc:511] Master 127.27.100.190:35397 requested a full tablet report, sending...
W20240507 08:05:57.381954 29110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:57.382889 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:57.383097 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:57.383249 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069157383232 us; error 0 us; skew 500 ppm
I20240507 08:05:57.383478 28902 ts_manager.cc:194] Registered new tserver with Master: cdca223a8c254da8b83a7786a86ae2be (127.27.100.130:41547)
I20240507 08:05:57.383847 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:57.384779 28902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59570
I20240507 08:05:57.386361 28050 webserver.cc:460] Webserver started at http://127.27.100.131:42147/ using document root <none> and password file <none>
I20240507 08:05:57.386803 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:57.386965 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:57.387157 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:57.388211 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-2-root/instance:
uuid: "ba8a6e4952f446f195bf407a1d708d49"
format_stamp: "Formatted at 2024-05-07 08:05:57 on dist-test-slave-j3cx"
I20240507 08:05:57.392146 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240507 08:05:57.395001 29115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:57.395694 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240507 08:05:57.395917 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-2-root
uuid: "ba8a6e4952f446f195bf407a1d708d49"
format_stamp: "Formatted at 2024-05-07 08:05:57 on dist-test-slave-j3cx"
I20240507 08:05:57.396163 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1715069146555183-28050-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:57.407917 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:05:57.408947 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:57.410208 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:05:57.412292 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:05:57.412472 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:57.412648 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:05:57.412770 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:57.476094 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.131:39945
I20240507 08:05:57.476176 29177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.131:39945 every 8 connection(s)
I20240507 08:05:57.489614 29178 heartbeater.cc:347] Connected to a master server at 127.27.100.190:35397
I20240507 08:05:57.489966 29178 heartbeater.cc:464] Registering TS with master...
I20240507 08:05:57.490689 29178 heartbeater.cc:511] Master 127.27.100.190:35397 requested a full tablet report, sending...
I20240507 08:05:57.492388 28902 ts_manager.cc:194] Registered new tserver with Master: ba8a6e4952f446f195bf407a1d708d49 (127.27.100.131:39945)
I20240507 08:05:57.492492 28050 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012122694s
I20240507 08:05:57.493693 28902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59576
I20240507 08:05:57.516378 28902 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59578:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
  }
}
W20240507 08:05:57.517884 28902 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:05:57.552443 29143 tablet_service.cc:1467] Processing CreateTablet for tablet 2d183053390b4e449cbb44ffacf2dee9 (DEFAULT_TABLE table=test-table [id=4fcd65d3a4064c74a6cd49377f08daaf]), partition=
I20240507 08:05:57.553567 29143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d183053390b4e449cbb44ffacf2dee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:57.555326 29069 tablet_service.cc:1467] Processing CreateTablet for tablet 2d183053390b4e449cbb44ffacf2dee9 (DEFAULT_TABLE table=test-table [id=4fcd65d3a4064c74a6cd49377f08daaf]), partition=
I20240507 08:05:57.556241 29069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d183053390b4e449cbb44ffacf2dee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:57.556525 28994 tablet_service.cc:1467] Processing CreateTablet for tablet 2d183053390b4e449cbb44ffacf2dee9 (DEFAULT_TABLE table=test-table [id=4fcd65d3a4064c74a6cd49377f08daaf]), partition=
I20240507 08:05:57.557632 28994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d183053390b4e449cbb44ffacf2dee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:57.571748 29193 tablet_bootstrap.cc:492] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be: Bootstrap starting.
I20240507 08:05:57.571781 29194 tablet_bootstrap.cc:492] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49: Bootstrap starting.
I20240507 08:05:57.574210 29195 tablet_bootstrap.cc:492] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3: Bootstrap starting.
I20240507 08:05:57.578866 29194 tablet_bootstrap.cc:654] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:57.579826 29193 tablet_bootstrap.cc:654] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:57.580197 29195 tablet_bootstrap.cc:654] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:57.586747 29194 tablet_bootstrap.cc:492] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49: No bootstrap required, opened a new log
I20240507 08:05:57.587060 29193 tablet_bootstrap.cc:492] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be: No bootstrap required, opened a new log
I20240507 08:05:57.587257 29194 ts_tablet_manager.cc:1367] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49: Time spent bootstrapping tablet: real 0.016s	user 0.010s	sys 0.004s
I20240507 08:05:57.587555 29193 ts_tablet_manager.cc:1367] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be: Time spent bootstrapping tablet: real 0.017s	user 0.014s	sys 0.000s
I20240507 08:05:57.588055 29195 tablet_bootstrap.cc:492] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3: No bootstrap required, opened a new log
I20240507 08:05:57.588508 29195 ts_tablet_manager.cc:1367] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.009s
I20240507 08:05:57.590562 29193 raft_consensus.cc:348] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0cddd4e15cb42e886e4671143b41ff3" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36845 } } peers { permanent_uuid: "ba8a6e4952f446f195bf407a1d708d49" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39945 } } peers { permanent_uuid: "cdca223a8c254da8b83a7786a86ae2be" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 41547 } }
I20240507 08:05:57.590723 29194 raft_consensus.cc:348] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0cddd4e15cb42e886e4671143b41ff3" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36845 } } peers { permanent_uuid: "ba8a6e4952f446f195bf407a1d708d49" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39945 } } peers { permanent_uuid: "cdca223a8c254da8b83a7786a86ae2be" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 41547 } }
I20240507 08:05:57.590853 29195 raft_consensus.cc:348] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0cddd4e15cb42e886e4671143b41ff3" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36845 } } peers { permanent_uuid: "ba8a6e4952f446f195bf407a1d708d49" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39945 } } peers { permanent_uuid: "cdca223a8c254da8b83a7786a86ae2be" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 41547 } }
I20240507 08:05:57.591465 29194 raft_consensus.cc:374] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:57.591678 29195 raft_consensus.cc:374] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:57.591432 29193 raft_consensus.cc:374] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:57.592044 29195 raft_consensus.cc:729] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0cddd4e15cb42e886e4671143b41ff3, State: Initialized, Role: FOLLOWER
I20240507 08:05:57.591985 29194 raft_consensus.cc:729] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8a6e4952f446f195bf407a1d708d49, State: Initialized, Role: FOLLOWER
I20240507 08:05:57.592128 29193 raft_consensus.cc:729] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdca223a8c254da8b83a7786a86ae2be, State: Initialized, Role: FOLLOWER
I20240507 08:05:57.593112 29193 consensus_queue.cc:260] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0cddd4e15cb42e886e4671143b41ff3" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36845 } } peers { permanent_uuid: "ba8a6e4952f446f195bf407a1d708d49" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39945 } } peers { permanent_uuid: "cdca223a8c254da8b83a7786a86ae2be" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 41547 } }
I20240507 08:05:57.592924 29194 consensus_queue.cc:260] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0cddd4e15cb42e886e4671143b41ff3" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36845 } } peers { permanent_uuid: "ba8a6e4952f446f195bf407a1d708d49" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39945 } } peers { permanent_uuid: "cdca223a8c254da8b83a7786a86ae2be" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 41547 } }
I20240507 08:05:57.592934 29195 consensus_queue.cc:260] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0cddd4e15cb42e886e4671143b41ff3" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36845 } } peers { permanent_uuid: "ba8a6e4952f446f195bf407a1d708d49" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39945 } } peers { permanent_uuid: "cdca223a8c254da8b83a7786a86ae2be" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 41547 } }
I20240507 08:05:57.598337 29029 heartbeater.cc:503] Master 127.27.100.190:35397 was elected leader, sending a full tablet report...
I20240507 08:05:57.600878 29195 ts_tablet_manager.cc:1398] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3: Time spent starting tablet: real 0.012s	user 0.000s	sys 0.007s
I20240507 08:05:57.602617 29104 heartbeater.cc:503] Master 127.27.100.190:35397 was elected leader, sending a full tablet report...
I20240507 08:05:57.602798 29178 heartbeater.cc:503] Master 127.27.100.190:35397 was elected leader, sending a full tablet report...
I20240507 08:05:57.604027 29194 ts_tablet_manager.cc:1398] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.007s
I20240507 08:05:57.605053 29193 ts_tablet_manager.cc:1398] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20240507 08:05:57.967449 29199 raft_consensus.cc:482] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:57.967926 29199 raft_consensus.cc:504] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0cddd4e15cb42e886e4671143b41ff3" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36845 } } peers { permanent_uuid: "ba8a6e4952f446f195bf407a1d708d49" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39945 } } peers { permanent_uuid: "cdca223a8c254da8b83a7786a86ae2be" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 41547 } }
I20240507 08:05:57.969959 29199 leader_election.cc:288] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0cddd4e15cb42e886e4671143b41ff3 (127.27.100.129:36845), ba8a6e4952f446f195bf407a1d708d49 (127.27.100.131:39945)
I20240507 08:05:57.980041 29153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d183053390b4e449cbb44ffacf2dee9" candidate_uuid: "cdca223a8c254da8b83a7786a86ae2be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8a6e4952f446f195bf407a1d708d49" is_pre_election: true
I20240507 08:05:57.980703 29004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d183053390b4e449cbb44ffacf2dee9" candidate_uuid: "cdca223a8c254da8b83a7786a86ae2be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0cddd4e15cb42e886e4671143b41ff3" is_pre_election: true
I20240507 08:05:57.980902 29153 raft_consensus.cc:2454] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdca223a8c254da8b83a7786a86ae2be in term 0.
I20240507 08:05:57.981230 29004 raft_consensus.cc:2454] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdca223a8c254da8b83a7786a86ae2be in term 0.
I20240507 08:05:57.981918 29042 leader_election.cc:302] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba8a6e4952f446f195bf407a1d708d49, cdca223a8c254da8b83a7786a86ae2be; no voters: 
I20240507 08:05:57.982666 29199 raft_consensus.cc:2781] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:57.983006 29199 raft_consensus.cc:482] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:57.983225 29199 raft_consensus.cc:3037] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:57.987732 29199 raft_consensus.cc:504] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0cddd4e15cb42e886e4671143b41ff3" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36845 } } peers { permanent_uuid: "ba8a6e4952f446f195bf407a1d708d49" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39945 } } peers { permanent_uuid: "cdca223a8c254da8b83a7786a86ae2be" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 41547 } }
I20240507 08:05:57.989279 29199 leader_election.cc:288] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [CANDIDATE]: Term 1 election: Requested vote from peers b0cddd4e15cb42e886e4671143b41ff3 (127.27.100.129:36845), ba8a6e4952f446f195bf407a1d708d49 (127.27.100.131:39945)
I20240507 08:05:57.990018 29004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d183053390b4e449cbb44ffacf2dee9" candidate_uuid: "cdca223a8c254da8b83a7786a86ae2be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0cddd4e15cb42e886e4671143b41ff3"
I20240507 08:05:57.990193 29153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d183053390b4e449cbb44ffacf2dee9" candidate_uuid: "cdca223a8c254da8b83a7786a86ae2be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8a6e4952f446f195bf407a1d708d49"
I20240507 08:05:57.990482 29004 raft_consensus.cc:3037] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:57.990680 29153 raft_consensus.cc:3037] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:57.995031 29004 raft_consensus.cc:2454] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdca223a8c254da8b83a7786a86ae2be in term 1.
I20240507 08:05:57.995033 29153 raft_consensus.cc:2454] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdca223a8c254da8b83a7786a86ae2be in term 1.
I20240507 08:05:57.996186 29044 leader_election.cc:302] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0cddd4e15cb42e886e4671143b41ff3, cdca223a8c254da8b83a7786a86ae2be; no voters: 
I20240507 08:05:57.996896 29199 raft_consensus.cc:2781] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:57.997738 29199 raft_consensus.cc:686] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [term 1 LEADER]: Becoming Leader. State: Replica: cdca223a8c254da8b83a7786a86ae2be, State: Running, Role: LEADER
I20240507 08:05:57.998534 29199 consensus_queue.cc:237] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0cddd4e15cb42e886e4671143b41ff3" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36845 } } peers { permanent_uuid: "ba8a6e4952f446f195bf407a1d708d49" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39945 } } peers { permanent_uuid: "cdca223a8c254da8b83a7786a86ae2be" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 41547 } }
I20240507 08:05:58.004644 28901 catalog_manager.cc:5484] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be reported cstate change: term changed from 0 to 1, leader changed from <none> to cdca223a8c254da8b83a7786a86ae2be (127.27.100.130). New cstate: current_term: 1 leader_uuid: "cdca223a8c254da8b83a7786a86ae2be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0cddd4e15cb42e886e4671143b41ff3" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba8a6e4952f446f195bf407a1d708d49" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdca223a8c254da8b83a7786a86ae2be" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 41547 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:58.437114 29199 consensus_queue.cc:1035] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0cddd4e15cb42e886e4671143b41ff3" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:58.447484 29211 consensus_queue.cc:1035] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8a6e4952f446f195bf407a1d708d49" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:59.070533 28050 tablet_replica.cc:330] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3: stopping tablet replica
I20240507 08:05:59.071061 28050 raft_consensus.cc:2229] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:59.071456 28050 raft_consensus.cc:2258] T 2d183053390b4e449cbb44ffacf2dee9 P b0cddd4e15cb42e886e4671143b41ff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:59.419430 29044 consensus_queue.cc:579] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [LEADER]: The tablet replica hosted on peer b0cddd4e15cb42e886e4671143b41ff3 has failed
W20240507 08:05:59.419886 29044 consensus_peers.cc:485] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be -> Peer b0cddd4e15cb42e886e4671143b41ff3 (127.27.100.129:36845): Couldn't send request to peer b0cddd4e15cb42e886e4671143b41ff3. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:59.595355 28050 tablet_server.cc:178] TabletServer@127.27.100.129:0 shutting down...
I20240507 08:05:59.611004 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:05:59.627463 28050 tablet_server.cc:194] TabletServer@127.27.100.129:0 shutdown complete.
I20240507 08:05:59.635401 28050 tablet_server.cc:178] TabletServer@127.27.100.130:0 shutting down...
I20240507 08:05:59.650099 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:05:59.650669 28050 tablet_replica.cc:330] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be: stopping tablet replica
I20240507 08:05:59.651147 28050 raft_consensus.cc:2229] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:05:59.651909 28050 raft_consensus.cc:2258] T 2d183053390b4e449cbb44ffacf2dee9 P cdca223a8c254da8b83a7786a86ae2be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:59.669270 28050 tablet_server.cc:194] TabletServer@127.27.100.130:0 shutdown complete.
I20240507 08:05:59.676558 28050 tablet_server.cc:178] TabletServer@127.27.100.131:0 shutting down...
I20240507 08:05:59.691394 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:05:59.692006 28050 tablet_replica.cc:330] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49: stopping tablet replica
I20240507 08:05:59.692516 28050 raft_consensus.cc:2229] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:59.692895 28050 raft_consensus.cc:2258] T 2d183053390b4e449cbb44ffacf2dee9 P ba8a6e4952f446f195bf407a1d708d49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:59.710196 28050 tablet_server.cc:194] TabletServer@127.27.100.131:0 shutdown complete.
I20240507 08:05:59.717324 28050 master.cc:537] Master@127.27.100.190:35397 shutting down...
I20240507 08:05:59.729743 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:05:59.730245 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:59.730532 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2b9b8fdba1934aba8c13894aba1998fd: stopping tablet replica
I20240507 08:05:59.746902 28050 master.cc:552] Master@127.27.100.190:35397 shutdown complete.
[       OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (2765 ms)
[ RUN      ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20240507 08:05:59.770037 28050 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.100.190:37779
I20240507 08:05:59.771023 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:59.776895 29222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:59.777844 29223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:59.778666 29225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:59.779557 28050 server_base.cc:1034] running on GCE node
I20240507 08:05:59.780215 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:59.780401 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:59.780550 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069159780529 us; error 0 us; skew 500 ppm
I20240507 08:05:59.781035 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:59.788969 28050 webserver.cc:460] Webserver started at http://127.27.100.190:44747/ using document root <none> and password file <none>
I20240507 08:05:59.789417 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:59.789590 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:59.789827 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:59.790848 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1715069146555183-28050-0/minicluster-data/master-0-root/instance:
uuid: "428d6cda5bc94cc78731dba21efc74f7"
format_stamp: "Formatted at 2024-05-07 08:05:59 on dist-test-slave-j3cx"
I20240507 08:05:59.794972 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240507 08:05:59.798055 29230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:59.798753 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240507 08:05:59.799010 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1715069146555183-28050-0/minicluster-data/master-0-root
uuid: "428d6cda5bc94cc78731dba21efc74f7"
format_stamp: "Formatted at 2024-05-07 08:05:59 on dist-test-slave-j3cx"
I20240507 08:05:59.799273 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1715069146555183-28050-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1715069146555183-28050-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1715069146555183-28050-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:59.827610 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:05:59.828624 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:59.863677 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.190:37779
I20240507 08:05:59.863763 29281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.190:37779 every 8 connection(s)
I20240507 08:05:59.867172 29282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:59.876433 29282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7: Bootstrap starting.
I20240507 08:05:59.880081 29282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:59.883632 29282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7: No bootstrap required, opened a new log
I20240507 08:05:59.885347 29282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428d6cda5bc94cc78731dba21efc74f7" member_type: VOTER }
I20240507 08:05:59.885720 29282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:59.885934 29282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428d6cda5bc94cc78731dba21efc74f7, State: Initialized, Role: FOLLOWER
I20240507 08:05:59.886437 29282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428d6cda5bc94cc78731dba21efc74f7" member_type: VOTER }
I20240507 08:05:59.886941 29282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:05:59.887147 29282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:05:59.887382 29282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:59.891250 29282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428d6cda5bc94cc78731dba21efc74f7" member_type: VOTER }
I20240507 08:05:59.891746 29282 leader_election.cc:302] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 428d6cda5bc94cc78731dba21efc74f7; no voters: 
I20240507 08:05:59.892784 29282 leader_election.cc:288] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:05:59.893060 29285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:59.894341 29285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [term 1 LEADER]: Becoming Leader. State: Replica: 428d6cda5bc94cc78731dba21efc74f7, State: Running, Role: LEADER
I20240507 08:05:59.894953 29285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428d6cda5bc94cc78731dba21efc74f7" member_type: VOTER }
I20240507 08:05:59.895437 29282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:05:59.897733 29287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 428d6cda5bc94cc78731dba21efc74f7. Latest consensus state: current_term: 1 leader_uuid: "428d6cda5bc94cc78731dba21efc74f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428d6cda5bc94cc78731dba21efc74f7" member_type: VOTER } }
I20240507 08:05:59.897703 29286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "428d6cda5bc94cc78731dba21efc74f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428d6cda5bc94cc78731dba21efc74f7" member_type: VOTER } }
I20240507 08:05:59.898424 29287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [sys.catalog]: This master's current role is: LEADER
I20240507 08:05:59.898526 29286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [sys.catalog]: This master's current role is: LEADER
I20240507 08:05:59.901638 29290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:05:59.909112 29290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:05:59.910014 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:05:59.916822 29290 catalog_manager.cc:1347] Generated new cluster ID: e3fb32acbc2342c19ef89b647e2dcece
I20240507 08:05:59.917089 29290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:05:59.932893 29290 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:05:59.934096 29290 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:05:59.949231 29290 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7: Generated new TSK 0
I20240507 08:05:59.949844 29290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:05:59.976382 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:59.982964 29303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:59.983884 29304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:59.984851 29306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:59.985716 28050 server_base.cc:1034] running on GCE node
I20240507 08:05:59.986450 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:59.986624 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:59.986781 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069159986763 us; error 0 us; skew 500 ppm
I20240507 08:05:59.987366 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:59.989670 28050 webserver.cc:460] Webserver started at http://127.27.100.129:35633/ using document root <none> and password file <none>
I20240507 08:05:59.990128 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:59.990298 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:59.990540 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:59.991611 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1715069146555183-28050-0/minicluster-data/ts-0-root/instance:
uuid: "0891ff8a3c84429784e066513cca7af5"
format_stamp: "Formatted at 2024-05-07 08:05:59 on dist-test-slave-j3cx"
I20240507 08:05:59.996052 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240507 08:05:59.999109 29311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:05:59.999876 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240507 08:06:00.000146 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "0891ff8a3c84429784e066513cca7af5"
format_stamp: "Formatted at 2024-05-07 08:05:59 on dist-test-slave-j3cx"
I20240507 08:06:00.000418 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:00.010733 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:00.011822 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:00.013121 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:00.015255 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:00.015441 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:00.015729 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:00.015882 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:00.052131 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:38753
I20240507 08:06:00.052217 29373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:38753 every 8 connection(s)
I20240507 08:06:00.065462 29374 heartbeater.cc:347] Connected to a master server at 127.27.100.190:37779
I20240507 08:06:00.065822 29374 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:00.066480 29374 heartbeater.cc:511] Master 127.27.100.190:37779 requested a full tablet report, sending...
I20240507 08:06:00.068332 29247 ts_manager.cc:194] Registered new tserver with Master: 0891ff8a3c84429784e066513cca7af5 (127.27.100.129:38753)
I20240507 08:06:00.069285 28050 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.014030025s
I20240507 08:06:00.069978 29247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40712
I20240507 08:06:00.088784 28050 tablet_server.cc:178] TabletServer@127.27.100.129:0 shutting down...
I20240507 08:06:00.105916 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:00.120990 28050 tablet_server.cc:194] TabletServer@127.27.100.129:0 shutdown complete.
I20240507 08:06:00.128158 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 08:06:00.130605 28050 tablet_server.cc:93] deduplicated master addresses: 127.27.100.190:37779
W20240507 08:06:00.134850 29384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:00.135726 29385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:00.136662 29387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:00.137639 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:00.138343 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:00.138515 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:00.138638 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069160138627 us; error 0 us; skew 500 ppm
I20240507 08:06:00.139094 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:00.141443 28050 webserver.cc:460] Webserver started at http://127.27.100.129:35633/ using document root <none> and password file <none>
I20240507 08:06:00.141870 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:00.142035 28050 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240507 08:06:00.145406 28050 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240507 08:06:00.147804 29392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:00.148492 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240507 08:06:00.148728 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "0891ff8a3c84429784e066513cca7af5"
format_stamp: "Formatted at 2024-05-07 08:05:59 on dist-test-slave-j3cx"
I20240507 08:06:00.148980 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:00.162353 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:00.163276 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:00.164603 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:00.166663 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:00.166831 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:00.167016 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:00.167138 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:00.277475 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:38753
I20240507 08:06:00.277573 29454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:38753 every 8 connection(s)
I20240507 08:06:00.284633 28050 tablet_server.cc:178] TabletServer@127.27.100.129:38753 shutting down...
I20240507 08:06:00.318557 29455 heartbeater.cc:347] Connected to a master server at 127.27.100.190:37779
I20240507 08:06:00.318998 29455 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:00.319715 29455 heartbeater.cc:511] Master 127.27.100.190:37779 requested a full tablet report, sending...
I20240507 08:06:00.321357 29247 ts_manager.cc:194] Re-registered known tserver with Master: 0891ff8a3c84429784e066513cca7af5 (127.27.100.129:38753)
I20240507 08:06:00.322623 29247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40740
I20240507 08:06:00.324697 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:00.339952 28050 tablet_server.cc:194] TabletServer@127.27.100.129:38753 shutdown complete.
I20240507 08:06:00.346714 28050 master.cc:537] Master@127.27.100.190:37779 shutting down...
I20240507 08:06:00.360673 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:00.361106 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:00.361434 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P 428d6cda5bc94cc78731dba21efc74f7: stopping tablet replica
W20240507 08:06:01.274883 29278 debug-util.cc:398] Leaking SignalData structure 0x7b0800036a60 after lost signal to thread 28053
W20240507 08:06:01.377401 28050 thread.cc:536] Waited for 1000ms trying to join with diag-logger (tid 29278)
W20240507 08:06:02.377883 28050 thread.cc:536] Waited for 2000ms trying to join with diag-logger (tid 29278)
I20240507 08:06:03.027956 28050 master.cc:552] Master@127.27.100.190:37779 shutdown complete.
[       OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (3280 ms)
[ RUN      ] TsTabletManagerITest.TestTableStats
I20240507 08:06:03.050379 28050 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.100.190:33177,127.27.100.189:36005,127.27.100.188:35355
I20240507 08:06:03.051604 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:03.058153 29462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:03.059094 29463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:03.060034 29465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:03.061082 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:03.062865 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:03.063068 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:03.063222 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069163063204 us; error 0 us; skew 500 ppm
I20240507 08:06:03.063776 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:03.066108 28050 webserver.cc:460] Webserver started at http://127.27.100.190:41647/ using document root <none> and password file <none>
I20240507 08:06:03.066547 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:03.066715 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:03.066964 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:03.068048 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-0-root/instance:
uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33"
format_stamp: "Formatted at 2024-05-07 08:06:03 on dist-test-slave-j3cx"
I20240507 08:06:03.072154 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.003s
I20240507 08:06:03.075116 29470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:03.075824 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240507 08:06:03.076092 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-0-root
uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33"
format_stamp: "Formatted at 2024-05-07 08:06:03 on dist-test-slave-j3cx"
I20240507 08:06:03.076346 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:03.102020 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:03.103015 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:03.137038 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.190:33177
I20240507 08:06:03.137130 29521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.190:33177 every 8 connection(s)
I20240507 08:06:03.140543 29522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:03.140784 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:03.147313 29524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:03.147562 29522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } has no permanent_uuid. Determining permanent_uuid...
W20240507 08:06:03.148475 29525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:03.149874 29527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:03.151016 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:03.151934 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:03.152135 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:03.152354 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069163152335 us; error 0 us; skew 500 ppm
I20240507 08:06:03.153062 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:03.155860 28050 webserver.cc:460] Webserver started at http://127.27.100.189:36919/ using document root <none> and password file <none>
I20240507 08:06:03.156307 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:03.156486 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:03.156718 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:03.157706 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-1-root/instance:
uuid: "66b1fdcb58294d68ad651ac2fa49acb5"
format_stamp: "Formatted at 2024-05-07 08:06:03 on dist-test-slave-j3cx"
I20240507 08:06:03.161047 29522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } has no permanent_uuid. Determining permanent_uuid...
I20240507 08:06:03.162629 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.005s
W20240507 08:06:03.163125 29471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.100.189:36005: connect: Connection refused (error 111)
W20240507 08:06:03.166100 29522 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.100.189:36005: Network error: Client connection negotiation failed: client connection to 127.27.100.189:36005: connect: Connection refused (error 111)
I20240507 08:06:03.167014 29535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:03.167758 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240507 08:06:03.168032 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-1-root
uuid: "66b1fdcb58294d68ad651ac2fa49acb5"
format_stamp: "Formatted at 2024-05-07 08:06:03 on dist-test-slave-j3cx"
I20240507 08:06:03.168294 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:03.185523 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:03.186435 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:03.217595 29522 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } attempt: 1
I20240507 08:06:03.222003 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.189:36005
I20240507 08:06:03.222064 29586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.189:36005 every 8 connection(s)
I20240507 08:06:03.226218 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 08:06:03.226559 29588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:06:03.232580 29590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:03.234066 29591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:03.235949 29588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } has no permanent_uuid. Determining permanent_uuid...
I20240507 08:06:03.237987 29522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } has no permanent_uuid. Determining permanent_uuid...
W20240507 08:06:03.242435 29593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:03.243196 29522 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.100.188:35355: Network error: Client connection negotiation failed: client connection to 127.27.100.188:35355: connect: Connection refused (error 111)
I20240507 08:06:03.245728 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:03.246704 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:03.246963 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:03.247166 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069163247150 us; error 0 us; skew 500 ppm
I20240507 08:06:03.247804 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:03.250869 28050 webserver.cc:460] Webserver started at http://127.27.100.188:44477/ using document root <none> and password file <none>
I20240507 08:06:03.251474 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:03.251749 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:03.252064 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:03.252427 29588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } has no permanent_uuid. Determining permanent_uuid...
I20240507 08:06:03.253384 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-2-root/instance:
uuid: "f03eb788ff4a4820b31171b231de9d41"
format_stamp: "Formatted at 2024-05-07 08:06:03 on dist-test-slave-j3cx"
I20240507 08:06:03.258677 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240507 08:06:03.262527 29599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:03.262586 29588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } has no permanent_uuid. Determining permanent_uuid...
I20240507 08:06:03.263429 28050 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240507 08:06:03.263762 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-2-root
uuid: "f03eb788ff4a4820b31171b231de9d41"
format_stamp: "Formatted at 2024-05-07 08:06:03 on dist-test-slave-j3cx"
I20240507 08:06:03.264041 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240507 08:06:03.268497 29588 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.100.188:35355: Network error: Client connection negotiation failed: client connection to 127.27.100.188:35355: connect: Connection refused (error 111)
I20240507 08:06:03.278321 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:03.279309 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:03.287082 29588 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } attempt: 1
I20240507 08:06:03.288647 29522 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } attempt: 1
W20240507 08:06:03.292976 29588 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.100.188:35355: Network error: Client connection negotiation failed: client connection to 127.27.100.188:35355: connect: Connection refused (error 111)
W20240507 08:06:03.294055 29522 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.27.100.188:35355: Network error: Client connection negotiation failed: client connection to 127.27.100.188:35355: connect: Connection refused (error 111)
I20240507 08:06:03.316903 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.188:35355
I20240507 08:06:03.316982 29651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.188:35355 every 8 connection(s)
I20240507 08:06:03.319903 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:06:03.320550 29652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:03.325981 29652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } has no permanent_uuid. Determining permanent_uuid...
I20240507 08:06:03.334568 29522 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } attempt: 2
I20240507 08:06:03.335695 29652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } has no permanent_uuid. Determining permanent_uuid...
I20240507 08:06:03.345901 29652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } has no permanent_uuid. Determining permanent_uuid...
I20240507 08:06:03.347448 29588 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } attempt: 2
I20240507 08:06:03.353039 29522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33: Bootstrap starting.
I20240507 08:06:03.358234 29522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:03.366147 29652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: Bootstrap starting.
I20240507 08:06:03.367760 29522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33: No bootstrap required, opened a new log
I20240507 08:06:03.371048 29652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:03.370558 29522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:03.371435 29522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:03.371817 29522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b50ddd4dc5d4bcb84e234822fcc7f33, State: Initialized, Role: FOLLOWER
I20240507 08:06:03.372570 29522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:03.375083 29652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: No bootstrap required, opened a new log
I20240507 08:06:03.377084 29652 raft_consensus.cc:348] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:03.377430 29588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5: Bootstrap starting.
I20240507 08:06:03.377652 29652 raft_consensus.cc:374] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:03.377962 29652 raft_consensus.cc:729] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03eb788ff4a4820b31171b231de9d41, State: Initialized, Role: FOLLOWER
I20240507 08:06:03.378602 29652 consensus_queue.cc:260] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:03.380582 29659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:03.381762 29659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:03.382275 29522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:06:03.384425 29588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:03.384415 29660 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:03.385555 29660 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:03.386003 29652 sys_catalog.cc:564] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:06:03.391302 29588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5: No bootstrap required, opened a new log
I20240507 08:06:03.394515 29588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:03.395692 29588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:03.396181 29588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b1fdcb58294d68ad651ac2fa49acb5, State: Initialized, Role: FOLLOWER
I20240507 08:06:03.397197 29588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:03.399936 29672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:03.400803 29672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:03.403350 29588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:06:03.405509 29672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:03.406100 29672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:03.409879 29497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" is_pre_election: true
I20240507 08:06:03.410606 29672 leader_election.cc:288] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:03.411532 29627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03eb788ff4a4820b31171b231de9d41" is_pre_election: true
I20240507 08:06:03.412438 29627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b1fdcb58294d68ad651ac2fa49acb5 in term 0.
I20240507 08:06:03.414321 29539 leader_election.cc:302] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b1fdcb58294d68ad651ac2fa49acb5, f03eb788ff4a4820b31171b231de9d41; no voters: 
I20240507 08:06:03.415370 29672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:06:03.410607 29497 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b1fdcb58294d68ad651ac2fa49acb5 in term 0.
I20240507 08:06:03.415822 29672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:03.416175 29672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:06:03.423388 29679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 08:06:03.424594 29679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 08:06:03.424643 29672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:03.427983 29497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33"
I20240507 08:06:03.428588 29497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:03.428967 29627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03eb788ff4a4820b31171b231de9d41"
I20240507 08:06:03.429546 29627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:03.435901 29497 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b1fdcb58294d68ad651ac2fa49acb5 in term 1.
I20240507 08:06:03.436434 29627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b1fdcb58294d68ad651ac2fa49acb5 in term 1.
I20240507 08:06:03.437290 29538 leader_election.cc:302] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b1fdcb58294d68ad651ac2fa49acb5, 8b50ddd4dc5d4bcb84e234822fcc7f33; no voters: 
I20240507 08:06:03.437844 29672 leader_election.cc:288] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 1 election: Requested vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:03.439016 29672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:03.439632 29672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 1 LEADER]: Becoming Leader. State: Replica: 66b1fdcb58294d68ad651ac2fa49acb5, State: Running, Role: LEADER
I20240507 08:06:03.441195 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240507 08:06:03.440505 29672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
W20240507 08:06:03.446239 29690 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 08:06:03.446846 29690 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 08:06:03.447037 29689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b1fdcb58294d68ad651ac2fa49acb5. Latest consensus state: current_term: 1 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:03.447906 29689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:03.449793 29696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:06:03.451133 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240507 08:06:03.453068 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 08:06:03.456055 29696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20240507 08:06:03.462327 29699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:03.463311 29700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:03.464358 29702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:03.470125 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:03.470259 29497 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 1 FOLLOWER]: Refusing update from remote peer 66b1fdcb58294d68ad651ac2fa49acb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:06:03.470388 29627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 1 FOLLOWER]: Refusing update from remote peer 66b1fdcb58294d68ad651ac2fa49acb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:06:03.471148 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:03.471448 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:03.471695 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069163471674 us; error 0 us; skew 500 ppm
I20240507 08:06:03.471858 29689 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:06:03.472570 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:03.472538 29672 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:06:03.475751 28050 webserver.cc:460] Webserver started at http://127.27.100.129:46699/ using document root <none> and password file <none>
I20240507 08:06:03.476238 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:03.476521 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:03.476822 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:03.478027 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-0-root/instance:
uuid: "3cebffdfceb24f75ae9450fce8382fa5"
format_stamp: "Formatted at 2024-05-07 08:06:03 on dist-test-slave-j3cx"
I20240507 08:06:03.483433 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240507 08:06:03.487066 29711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:03.488016 28050 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240507 08:06:03.488420 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "3cebffdfceb24f75ae9450fce8382fa5"
format_stamp: "Formatted at 2024-05-07 08:06:03 on dist-test-slave-j3cx"
I20240507 08:06:03.488802 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:03.489270 29660 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b1fdcb58294d68ad651ac2fa49acb5. Latest consensus state: current_term: 1 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:03.489992 29660 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:03.492967 29659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b1fdcb58294d68ad651ac2fa49acb5. Latest consensus state: current_term: 1 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:03.493721 29659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:03.499855 29708 mvcc.cc:205] Tried to move back new op lower bound from 7024923293560528896 to 7024923293461319680. Current Snapshot: MvccSnapshot[applied={T|T < 7024923293560528896}]
I20240507 08:06:03.512143 29660 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:03.513202 29660 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:03.516942 29659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:03.517640 29659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:03.521462 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:03.522504 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:03.523093 29696 catalog_manager.cc:1347] Generated new cluster ID: a5e9cfaea5f8455aa6e905b407ec60c8
I20240507 08:06:03.523402 29696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:06:03.524340 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:03.527657 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:03.527931 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:03.528198 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:03.528416 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:03.528051 29672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:03.528678 29672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:03.529292 29689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:03.529901 29689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:03.565265 29696 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:06:03.567524 29696 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:06:03.594977 29696 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5: Generated new TSK 0
I20240507 08:06:03.595702 29696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:06:03.607301 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:35455
I20240507 08:06:03.607389 29779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:35455 every 8 connection(s)
I20240507 08:06:03.635203 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 08:06:03.637427 29780 heartbeater.cc:347] Connected to a master server at 127.27.100.188:35355
I20240507 08:06:03.637796 29780 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:03.637956 29781 heartbeater.cc:347] Connected to a master server at 127.27.100.189:36005
I20240507 08:06:03.638329 29781 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:03.638746 29780 heartbeater.cc:511] Master 127.27.100.188:35355 requested a full tablet report, sending...
I20240507 08:06:03.639125 29781 heartbeater.cc:511] Master 127.27.100.189:36005 requested a full tablet report, sending...
I20240507 08:06:03.641391 29552 ts_manager.cc:194] Registered new tserver with Master: 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455)
I20240507 08:06:03.642213 29617 ts_manager.cc:194] Registered new tserver with Master: 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455)
I20240507 08:06:03.643990 29552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36304
I20240507 08:06:03.646406 29782 heartbeater.cc:347] Connected to a master server at 127.27.100.190:33177
I20240507 08:06:03.646788 29782 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:03.647591 29782 heartbeater.cc:511] Master 127.27.100.190:33177 requested a full tablet report, sending...
W20240507 08:06:03.650213 29787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:03.650399 29487 ts_manager.cc:194] Registered new tserver with Master: 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455)
W20240507 08:06:03.651610 29788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:03.658767 28050 server_base.cc:1034] running on GCE node
W20240507 08:06:03.659807 29791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:03.660601 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:03.660820 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:03.660991 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069163660979 us; error 0 us; skew 500 ppm
I20240507 08:06:03.661514 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:03.663815 28050 webserver.cc:460] Webserver started at http://127.27.100.130:35845/ using document root <none> and password file <none>
I20240507 08:06:03.664273 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:03.664459 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:03.664710 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:03.665756 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-1-root/instance:
uuid: "3f397b9469554143884dc544bf64282a"
format_stamp: "Formatted at 2024-05-07 08:06:03 on dist-test-slave-j3cx"
I20240507 08:06:03.669996 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.006s
I20240507 08:06:03.673382 29795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:03.674022 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240507 08:06:03.674293 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-1-root
uuid: "3f397b9469554143884dc544bf64282a"
format_stamp: "Formatted at 2024-05-07 08:06:03 on dist-test-slave-j3cx"
I20240507 08:06:03.674544 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:03.711686 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:03.712921 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:03.714804 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:03.717542 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:03.717718 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:03.717952 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:03.718108 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:03.747098 29781 heartbeater.cc:503] Master 127.27.100.189:36005 was elected leader, sending a full tablet report...
I20240507 08:06:03.793030 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.130:36791
I20240507 08:06:03.793118 29857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.130:36791 every 8 connection(s)
I20240507 08:06:03.813894 29858 heartbeater.cc:347] Connected to a master server at 127.27.100.188:35355
I20240507 08:06:03.814429 29858 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:03.815454 29858 heartbeater.cc:511] Master 127.27.100.188:35355 requested a full tablet report, sending...
I20240507 08:06:03.816540 29859 heartbeater.cc:347] Connected to a master server at 127.27.100.189:36005
I20240507 08:06:03.817061 29859 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:03.818161 29617 ts_manager.cc:194] Registered new tserver with Master: 3f397b9469554143884dc544bf64282a (127.27.100.130:36791)
I20240507 08:06:03.818235 29859 heartbeater.cc:511] Master 127.27.100.189:36005 requested a full tablet report, sending...
I20240507 08:06:03.821116 29552 ts_manager.cc:194] Registered new tserver with Master: 3f397b9469554143884dc544bf64282a (127.27.100.130:36791)
I20240507 08:06:03.821316 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 08:06:03.823395 29552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36306
I20240507 08:06:03.828783 29861 heartbeater.cc:347] Connected to a master server at 127.27.100.190:33177
I20240507 08:06:03.829228 29861 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:03.830013 29861 heartbeater.cc:511] Master 127.27.100.190:33177 requested a full tablet report, sending...
W20240507 08:06:03.831393 29865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:03.833422 29487 ts_manager.cc:194] Registered new tserver with Master: 3f397b9469554143884dc544bf64282a (127.27.100.130:36791)
W20240507 08:06:03.835253 29866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:03.836045 29868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:03.837349 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:03.838048 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:03.838260 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:03.838449 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069163838434 us; error 0 us; skew 500 ppm
I20240507 08:06:03.838954 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:03.841373 28050 webserver.cc:460] Webserver started at http://127.27.100.131:42833/ using document root <none> and password file <none>
I20240507 08:06:03.841835 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:03.842015 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:03.842254 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:03.843303 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-2-root/instance:
uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f"
format_stamp: "Formatted at 2024-05-07 08:06:03 on dist-test-slave-j3cx"
I20240507 08:06:03.847570 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240507 08:06:03.850728 29873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:03.851569 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.003s
I20240507 08:06:03.851846 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-2-root
uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f"
format_stamp: "Formatted at 2024-05-07 08:06:03 on dist-test-slave-j3cx"
I20240507 08:06:03.852156 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:03.866314 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:03.867444 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:03.868909 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:03.870869 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:03.871073 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:03.871380 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:03.871639 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20240507 08:06:03.927168 29859 heartbeater.cc:503] Master 127.27.100.189:36005 was elected leader, sending a full tablet report...
I20240507 08:06:03.946985 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.131:44631
I20240507 08:06:03.947089 29935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.131:44631 every 8 connection(s)
I20240507 08:06:03.971771 29936 heartbeater.cc:347] Connected to a master server at 127.27.100.188:35355
I20240507 08:06:03.972222 29936 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:03.973202 29936 heartbeater.cc:511] Master 127.27.100.188:35355 requested a full tablet report, sending...
I20240507 08:06:03.974695 29939 heartbeater.cc:347] Connected to a master server at 127.27.100.190:33177
I20240507 08:06:03.975106 29939 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:03.975392 29937 heartbeater.cc:347] Connected to a master server at 127.27.100.189:36005
I20240507 08:06:03.975804 29937 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:03.975786 29617 ts_manager.cc:194] Registered new tserver with Master: 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:03.976168 29939 heartbeater.cc:511] Master 127.27.100.190:33177 requested a full tablet report, sending...
I20240507 08:06:03.976934 29937 heartbeater.cc:511] Master 127.27.100.189:36005 requested a full tablet report, sending...
I20240507 08:06:03.978904 29552 ts_manager.cc:194] Registered new tserver with Master: 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:03.979238 29487 ts_manager.cc:194] Registered new tserver with Master: 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:03.979974 28050 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016084411s
I20240507 08:06:03.980643 29552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36322
I20240507 08:06:04.041405 29552 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36328:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240507 08:06:04.043254 29552 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:06:04.083308 29937 heartbeater.cc:503] Master 127.27.100.189:36005 was elected leader, sending a full tablet report...
I20240507 08:06:04.133503 29822 tablet_service.cc:1467] Processing CreateTablet for tablet 55e5e59c50f54e548e29ec71cabf4e69 (DEFAULT_TABLE table=test-table [id=1d79b5e45252475d87377636deec0dd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:04.135108 29822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e5e59c50f54e548e29ec71cabf4e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:04.134985 29823 tablet_service.cc:1467] Processing CreateTablet for tablet 6eeffdf4714a4ad28391f055890b46ed (DEFAULT_TABLE table=test-table [id=1d79b5e45252475d87377636deec0dd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:04.136421 29823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eeffdf4714a4ad28391f055890b46ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:04.133446 29821 tablet_service.cc:1467] Processing CreateTablet for tablet 71c9b00c40c6458f9a0fe4a3a57fc99f (DEFAULT_TABLE table=test-table [id=1d79b5e45252475d87377636deec0dd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:04.139978 29821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c9b00c40c6458f9a0fe4a3a57fc99f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:04.141983 29744 tablet_service.cc:1467] Processing CreateTablet for tablet 6eeffdf4714a4ad28391f055890b46ed (DEFAULT_TABLE table=test-table [id=1d79b5e45252475d87377636deec0dd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:04.143764 29742 tablet_service.cc:1467] Processing CreateTablet for tablet 71c9b00c40c6458f9a0fe4a3a57fc99f (DEFAULT_TABLE table=test-table [id=1d79b5e45252475d87377636deec0dd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:04.139847 29743 tablet_service.cc:1467] Processing CreateTablet for tablet 55e5e59c50f54e548e29ec71cabf4e69 (DEFAULT_TABLE table=test-table [id=1d79b5e45252475d87377636deec0dd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:04.146999 29744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eeffdf4714a4ad28391f055890b46ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:04.148063 29743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e5e59c50f54e548e29ec71cabf4e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:04.149300 29742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c9b00c40c6458f9a0fe4a3a57fc99f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:04.154317 29901 tablet_service.cc:1467] Processing CreateTablet for tablet 6eeffdf4714a4ad28391f055890b46ed (DEFAULT_TABLE table=test-table [id=1d79b5e45252475d87377636deec0dd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:04.155601 29901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eeffdf4714a4ad28391f055890b46ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:04.158339 29900 tablet_service.cc:1467] Processing CreateTablet for tablet 55e5e59c50f54e548e29ec71cabf4e69 (DEFAULT_TABLE table=test-table [id=1d79b5e45252475d87377636deec0dd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:04.160315 29900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e5e59c50f54e548e29ec71cabf4e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:04.163659 29899 tablet_service.cc:1467] Processing CreateTablet for tablet 71c9b00c40c6458f9a0fe4a3a57fc99f (DEFAULT_TABLE table=test-table [id=1d79b5e45252475d87377636deec0dd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:04.166338 29899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c9b00c40c6458f9a0fe4a3a57fc99f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:04.183184 29957 tablet_bootstrap.cc:492] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5: Bootstrap starting.
I20240507 08:06:04.222601 29958 tablet_bootstrap.cc:492] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a: Bootstrap starting.
I20240507 08:06:04.231698 29959 tablet_bootstrap.cc:492] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f: Bootstrap starting.
I20240507 08:06:04.238456 29959 tablet_bootstrap.cc:654] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:04.241791 29958 tablet_bootstrap.cc:654] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:04.244114 29957 tablet_bootstrap.cc:654] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:04.253895 29959 tablet_bootstrap.cc:492] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f: No bootstrap required, opened a new log
I20240507 08:06:04.254016 29957 tablet_bootstrap.cc:492] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5: No bootstrap required, opened a new log
I20240507 08:06:04.254351 29959 ts_tablet_manager.cc:1367] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f: Time spent bootstrapping tablet: real 0.023s	user 0.013s	sys 0.003s
I20240507 08:06:04.254547 29957 ts_tablet_manager.cc:1367] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5: Time spent bootstrapping tablet: real 0.072s	user 0.009s	sys 0.006s
I20240507 08:06:04.256212 29958 tablet_bootstrap.cc:492] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a: No bootstrap required, opened a new log
I20240507 08:06:04.256690 29958 ts_tablet_manager.cc:1367] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a: Time spent bootstrapping tablet: real 0.035s	user 0.007s	sys 0.007s
I20240507 08:06:04.257076 29957 raft_consensus.cc:348] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:04.257665 29957 raft_consensus.cc:374] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:04.258142 29957 raft_consensus.cc:729] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cebffdfceb24f75ae9450fce8382fa5, State: Initialized, Role: FOLLOWER
I20240507 08:06:04.257644 29959 raft_consensus.cc:348] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.258603 29959 raft_consensus.cc:374] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:04.258955 29959 raft_consensus.cc:729] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64580d7ab4d84b86b6bb4a3e38fbd43f, State: Initialized, Role: FOLLOWER
I20240507 08:06:04.258778 29957 consensus_queue.cc:260] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:04.259559 29958 raft_consensus.cc:348] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:04.259785 29959 consensus_queue.cc:260] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.260365 29958 raft_consensus.cc:374] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:04.262121 29958 raft_consensus.cc:729] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f397b9469554143884dc544bf64282a, State: Initialized, Role: FOLLOWER
I20240507 08:06:04.263062 29957 ts_tablet_manager.cc:1398] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20240507 08:06:04.262823 29958 consensus_queue.cc:260] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:04.266624 29957 tablet_bootstrap.cc:492] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5: Bootstrap starting.
I20240507 08:06:04.274657 29957 tablet_bootstrap.cc:654] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:04.281972 29957 tablet_bootstrap.cc:492] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5: No bootstrap required, opened a new log
I20240507 08:06:04.282743 29957 ts_tablet_manager.cc:1367] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5: Time spent bootstrapping tablet: real 0.016s	user 0.010s	sys 0.004s
I20240507 08:06:04.286183 29957 raft_consensus.cc:348] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.287060 29957 raft_consensus.cc:374] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:04.287380 29957 raft_consensus.cc:729] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cebffdfceb24f75ae9450fce8382fa5, State: Initialized, Role: FOLLOWER
I20240507 08:06:04.288506 29959 ts_tablet_manager.cc:1398] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f: Time spent starting tablet: real 0.034s	user 0.002s	sys 0.012s
I20240507 08:06:04.288228 29957 consensus_queue.cc:260] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.289366 29958 ts_tablet_manager.cc:1398] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a: Time spent starting tablet: real 0.032s	user 0.016s	sys 0.012s
I20240507 08:06:04.290411 29958 tablet_bootstrap.cc:492] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a: Bootstrap starting.
I20240507 08:06:04.290928 29957 ts_tablet_manager.cc:1398] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.003s
I20240507 08:06:04.291950 29957 tablet_bootstrap.cc:492] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5: Bootstrap starting.
I20240507 08:06:04.296494 29958 tablet_bootstrap.cc:654] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:04.297641 29957 tablet_bootstrap.cc:654] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:04.302289 29959 tablet_bootstrap.cc:492] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f: Bootstrap starting.
I20240507 08:06:04.304687 29963 raft_consensus.cc:482] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.305377 29963 raft_consensus.cc:504] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.315604 29963 leader_election.cc:288] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:04.327210 29963 raft_consensus.cc:482] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.336032 29965 raft_consensus.cc:482] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.336678 29965 raft_consensus.cc:504] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:04.328311 29963 raft_consensus.cc:504] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:04.342962 29963 leader_election.cc:288] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:04.344620 29964 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.346381 29964 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.382182 29964 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 3f397b9469554143884dc544bf64282a (127.27.100.130:36791)
I20240507 08:06:04.397059 29959 tablet_bootstrap.cc:654] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:04.386886 29965 leader_election.cc:288] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631), 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455)
I20240507 08:06:04.429728 29958 tablet_bootstrap.cc:492] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a: No bootstrap required, opened a new log
I20240507 08:06:04.430380 29958 ts_tablet_manager.cc:1367] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a: Time spent bootstrapping tablet: real 0.140s	user 0.044s	sys 0.051s
I20240507 08:06:04.439878 29755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:04.447566 29679 catalog_manager.cc:1259] Loaded cluster ID: a5e9cfaea5f8455aa6e905b407ec60c8
I20240507 08:06:04.448221 29679 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: loading cluster ID for follower catalog manager: success
I20240507 08:06:04.456363 29690 catalog_manager.cc:1259] Loaded cluster ID: a5e9cfaea5f8455aa6e905b407ec60c8
I20240507 08:06:04.456804 29690 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33: loading cluster ID for follower catalog manager: success
W20240507 08:06:04.464304 29876 leader_election.cc:341] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:06:04.490767 29911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
W20240507 08:06:04.496387 29799 leader_election.cc:341] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:06:04.503170 29832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:04.503979 29832 raft_consensus.cc:2454] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cebffdfceb24f75ae9450fce8382fa5 in term 0.
I20240507 08:06:04.505702 29716 leader_election.cc:302] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5, 3f397b9469554143884dc544bf64282a; no voters: 
I20240507 08:06:04.506882 29963 raft_consensus.cc:2781] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:06:04.506426 29958 raft_consensus.cc:348] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.507297 29963 raft_consensus.cc:482] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:04.507548 29958 raft_consensus.cc:374] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:04.507763 29963 raft_consensus.cc:3037] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:04.508042 29958 raft_consensus.cc:729] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f397b9469554143884dc544bf64282a, State: Initialized, Role: FOLLOWER
I20240507 08:06:04.509042 29833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:04.509418 29957 tablet_bootstrap.cc:492] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5: No bootstrap required, opened a new log
I20240507 08:06:04.513717 29957 ts_tablet_manager.cc:1367] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5: Time spent bootstrapping tablet: real 0.222s	user 0.056s	sys 0.096s
I20240507 08:06:04.509910 29679 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: acquiring CA information for follower catalog manager: success
I20240507 08:06:04.531163 29911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:04.725495 29910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:04.509054 29958 consensus_queue.cc:260] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.741569 29959 tablet_bootstrap.cc:492] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f: No bootstrap required, opened a new log
I20240507 08:06:04.742559 29690 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33: acquiring CA information for follower catalog manager: success
W20240507 08:06:04.751102 29798 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.100.129:35455, user_credentials={real_user=slave}} blocked reactor thread for 163401us
I20240507 08:06:04.745702 29965 raft_consensus.cc:482] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.749644 29959 ts_tablet_manager.cc:1367] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f: Time spent bootstrapping tablet: real 0.448s	user 0.030s	sys 0.020s
I20240507 08:06:04.755721 29957 raft_consensus.cc:348] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.764590 29957 raft_consensus.cc:374] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:04.764947 29957 raft_consensus.cc:729] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cebffdfceb24f75ae9450fce8382fa5, State: Initialized, Role: FOLLOWER
I20240507 08:06:04.766086 29755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:04.766729 29958 ts_tablet_manager.cc:1398] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a: Time spent starting tablet: real 0.336s	user 0.007s	sys 0.002s
I20240507 08:06:04.760340 29959 raft_consensus.cc:348] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:04.762241 29965 raft_consensus.cc:504] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.773571 29978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66b1fdcb58294d68ad651ac2fa49acb5)
I20240507 08:06:04.782037 29978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:04.783656 29959 raft_consensus.cc:374] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:04.783282 29965 leader_election.cc:288] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:04.787231 29679 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240507 08:06:04.785362 29754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:04.787832 29754 raft_consensus.cc:2454] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f397b9469554143884dc544bf64282a in term 0.
I20240507 08:06:04.785882 29833 raft_consensus.cc:2454] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cebffdfceb24f75ae9450fce8382fa5 in term 0.
I20240507 08:06:04.787101 29911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:04.784197 29959 raft_consensus.cc:729] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64580d7ab4d84b86b6bb4a3e38fbd43f, State: Initialized, Role: FOLLOWER
W20240507 08:06:04.789763 29799 leader_election.cc:341] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:06:04.790611 29833 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45950 (request call id 0) took 301 ms (client timeout 263 ms). Trace:
I20240507 08:06:04.790230 29959 consensus_queue.cc:260] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
W20240507 08:06:04.791078 29833 rpcz_store.cc:269] 0507 08:06:04.492026 (+     0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:04.493626 (+  1600us) service_pool.cc:225] Handling call
0507 08:06:04.790453 (+296827us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54003502d4. stack: 00007f447ae7eba3 00007f447ae7eaa0 00007f447a7e3584 00000000003bd474 00000000003bd409 00007f44816a76b1 00007f44817264fe 00007f448392e9c7 00007f44812b7c14 00007f44812b7ba2 00007f44812b7ad1 00007f44812b7a4c 00007f44812b6d02 00007f447f9e3bcc 00007f447f9e3006 00007f447f9e29af
0507 08:06:04.790550 (+    97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":626799488}
I20240507 08:06:04.794139 29963 raft_consensus.cc:504] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:04.796778 29963 leader_election.cc:288] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 1 election: Requested vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:04.797410 29964 raft_consensus.cc:482] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.792824 29959 ts_tablet_manager.cc:1398] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f: Time spent starting tablet: real 0.035s	user 0.006s	sys 0.000s
I20240507 08:06:04.797755 29755 raft_consensus.cc:2379] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f397b9469554143884dc544bf64282a in current term 1: Already voted for candidate 3cebffdfceb24f75ae9450fce8382fa5 in this term.
I20240507 08:06:04.798629 29964 raft_consensus.cc:504] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:04.804265 29964 leader_election.cc:288] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455)
W20240507 08:06:04.805636 29877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.100.130:36791, user_credentials={real_user=slave}} blocked reactor thread for 59277.7us
I20240507 08:06:04.805897 29911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f"
I20240507 08:06:04.806155 29833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a"
I20240507 08:06:04.806511 29690 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240507 08:06:04.806767 29833 raft_consensus.cc:3037] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:04.806524 29911 raft_consensus.cc:3037] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:04.807669 29754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:04.809396 29832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:04.797552 29798 leader_election.cc:302] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5, 3f397b9469554143884dc544bf64282a; no voters: 64580d7ab4d84b86b6bb4a3e38fbd43f
I20240507 08:06:04.814693 29911 raft_consensus.cc:2454] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cebffdfceb24f75ae9450fce8382fa5 in term 1.
I20240507 08:06:04.811980 29831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:04.816072 29965 raft_consensus.cc:2781] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:06:04.815927 29877 leader_election.cc:302] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 3cebffdfceb24f75ae9450fce8382fa5, 3f397b9469554143884dc544bf64282a
I20240507 08:06:04.765619 29957 consensus_queue.cc:260] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.816485 29965 raft_consensus.cc:482] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:04.819327 29979 raft_consensus.cc:482] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66b1fdcb58294d68ad651ac2fa49acb5)
I20240507 08:06:04.820044 29965 raft_consensus.cc:3037] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:04.820590 29979 raft_consensus.cc:504] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:04.823189 29497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f03eb788ff4a4820b31171b231de9d41" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" is_pre_election: true
I20240507 08:06:04.825449 29979 leader_election.cc:288] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005)
I20240507 08:06:04.826059 29964 raft_consensus.cc:2726] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:06:04.816974 29798 leader_election.cc:302] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f397b9469554143884dc544bf64282a; no voters: 3cebffdfceb24f75ae9450fce8382fa5, 64580d7ab4d84b86b6bb4a3e38fbd43f
I20240507 08:06:04.826627 29562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f03eb788ff4a4820b31171b231de9d41" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" is_pre_election: true
I20240507 08:06:04.827039 29959 tablet_bootstrap.cc:492] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f: Bootstrap starting.
I20240507 08:06:04.819096 29958 tablet_bootstrap.cc:492] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a: Bootstrap starting.
I20240507 08:06:04.826483 29963 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.829052 29601 leader_election.cc:302] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f03eb788ff4a4820b31171b231de9d41; no voters: 66b1fdcb58294d68ad651ac2fa49acb5, 8b50ddd4dc5d4bcb84e234822fcc7f33
I20240507 08:06:04.829399 29716 leader_election.cc:302] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5, 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 
I20240507 08:06:04.829640 29831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:04.830567 29910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:04.831184 29910 raft_consensus.cc:2362] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3cebffdfceb24f75ae9450fce8382fa5 in term 1. Re-sending same reply.
I20240507 08:06:04.832405 29911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:04.834190 29963 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
W20240507 08:06:04.835114 29877 leader_election.cc:341] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f397b9469554143884dc544bf64282a (127.27.100.130:36791): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:06:04.835657 29877 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 3cebffdfceb24f75ae9450fce8382fa5, 3f397b9469554143884dc544bf64282a
I20240507 08:06:04.836570 29964 raft_consensus.cc:2726] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:06:04.837694 29963 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:04.815763 29833 raft_consensus.cc:2454] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cebffdfceb24f75ae9450fce8382fa5 in term 1.
I20240507 08:06:04.841883 29562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" is_pre_election: true
I20240507 08:06:04.852092 29970 raft_consensus.cc:2726] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:06:04.843806 29978 leader_election.cc:288] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:04.844249 29627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f03eb788ff4a4820b31171b231de9d41" is_pre_election: true
I20240507 08:06:04.848364 29983 raft_consensus.cc:2781] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:04.854867 29958 tablet_bootstrap.cc:654] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:04.855224 29474 leader_election.cc:302] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b50ddd4dc5d4bcb84e234822fcc7f33; no voters: 66b1fdcb58294d68ad651ac2fa49acb5, f03eb788ff4a4820b31171b231de9d41
I20240507 08:06:04.855244 29983 raft_consensus.cc:686] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 LEADER]: Becoming Leader. State: Replica: 3cebffdfceb24f75ae9450fce8382fa5, State: Running, Role: LEADER
I20240507 08:06:04.857617 29983 consensus_queue.cc:237] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:04.859122 29833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:04.861577 29911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:04.860685 29957 ts_tablet_manager.cc:1398] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5: Time spent starting tablet: real 0.347s	user 0.002s	sys 0.008s
I20240507 08:06:04.849798 29965 raft_consensus.cc:504] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.849557 29979 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:06:04.862378 29911 raft_consensus.cc:2454] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cebffdfceb24f75ae9450fce8382fa5 in term 0.
I20240507 08:06:04.863651 29831 raft_consensus.cc:2379] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3cebffdfceb24f75ae9450fce8382fa5 in current term 1: Already voted for candidate 3f397b9469554143884dc544bf64282a in this term.
I20240507 08:06:04.866621 29965 leader_election.cc:288] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 1 election: Requested vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:04.867887 29978 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:06:04.868647 29911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f"
W20240507 08:06:04.870458 29799 leader_election.cc:341] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:06:04.870810 29755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5"
I20240507 08:06:04.871589 29755 raft_consensus.cc:3037] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:04.876045 29959 tablet_bootstrap.cc:654] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f: Neither blocks nor log segments found. Creating new log.
W20240507 08:06:04.878132 29716 leader_election.cc:341] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3f397b9469554143884dc544bf64282a (127.27.100.130:36791): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:06:04.882766 29716 leader_election.cc:341] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:06:04.879065 29958 tablet_bootstrap.cc:492] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a: No bootstrap required, opened a new log
I20240507 08:06:04.879809 29755 raft_consensus.cc:2454] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f397b9469554143884dc544bf64282a in term 1.
I20240507 08:06:04.884871 29958 ts_tablet_manager.cc:1367] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a: Time spent bootstrapping tablet: real 0.066s	user 0.019s	sys 0.011s
I20240507 08:06:04.883463 29716 leader_election.cc:302] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5; no voters: 3f397b9469554143884dc544bf64282a, 64580d7ab4d84b86b6bb4a3e38fbd43f
I20240507 08:06:04.883746 29551 catalog_manager.cc:5484] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129). New cstate: current_term: 1 leader_uuid: "3cebffdfceb24f75ae9450fce8382fa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:04.885814 29798 leader_election.cc:302] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5, 3f397b9469554143884dc544bf64282a; no voters: 64580d7ab4d84b86b6bb4a3e38fbd43f
I20240507 08:06:04.888444 29965 raft_consensus.cc:2781] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:04.888557 29959 tablet_bootstrap.cc:492] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f: No bootstrap required, opened a new log
I20240507 08:06:04.889022 29965 raft_consensus.cc:686] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 1 LEADER]: Becoming Leader. State: Replica: 3f397b9469554143884dc544bf64282a, State: Running, Role: LEADER
I20240507 08:06:04.888881 29958 raft_consensus.cc:348] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.889731 29958 raft_consensus.cc:374] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:04.890062 29958 raft_consensus.cc:729] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f397b9469554143884dc544bf64282a, State: Initialized, Role: FOLLOWER
I20240507 08:06:04.889899 29965 consensus_queue.cc:237] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.891052 29983 raft_consensus.cc:2726] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:06:04.890903 29958 consensus_queue.cc:260] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.891454 29716 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5, 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 3f397b9469554143884dc544bf64282a
I20240507 08:06:04.891805 29959 ts_tablet_manager.cc:1367] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f: Time spent bootstrapping tablet: real 0.065s	user 0.014s	sys 0.000s
I20240507 08:06:04.893584 29958 ts_tablet_manager.cc:1398] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20240507 08:06:04.895610 29983 raft_consensus.cc:2781] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:06:04.895982 29983 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:04.896399 29983 raft_consensus.cc:3037] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:04.897874 29970 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.898361 29970 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.900735 29970 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:04.902729 29911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:04.903518 29911 raft_consensus.cc:2454] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f397b9469554143884dc544bf64282a in term 0.
I20240507 08:06:04.904857 29799 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f397b9469554143884dc544bf64282a, 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 
I20240507 08:06:04.905827 29755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:04.906533 29965 raft_consensus.cc:2781] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:06:04.906960 29965 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:04.908272 29965 raft_consensus.cc:3037] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:04.909986 29552 catalog_manager.cc:5484] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f397b9469554143884dc544bf64282a (127.27.100.130). New cstate: current_term: 1 leader_uuid: "3f397b9469554143884dc544bf64282a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:04.916363 29965 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.917093 29959 raft_consensus.cc:348] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.917912 29959 raft_consensus.cc:374] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:04.918772 29965 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 1 election: Requested vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:04.919404 29754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5"
I20240507 08:06:04.920717 29911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f"
I20240507 08:06:04.921444 29911 raft_consensus.cc:3037] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:04.918203 29959 raft_consensus.cc:729] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64580d7ab4d84b86b6bb4a3e38fbd43f, State: Initialized, Role: FOLLOWER
I20240507 08:06:04.933300 29983 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.934648 29959 consensus_queue.cc:260] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.942165 29755 raft_consensus.cc:2379] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f397b9469554143884dc544bf64282a in current term 1: Already voted for candidate 3cebffdfceb24f75ae9450fce8382fa5 in this term.
I20240507 08:06:04.941229 29959 ts_tablet_manager.cc:1398] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f: Time spent starting tablet: real 0.049s	user 0.008s	sys 0.000s
I20240507 08:06:04.943675 29831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a"
I20240507 08:06:04.944522 29831 raft_consensus.cc:2379] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3cebffdfceb24f75ae9450fce8382fa5 in current term 1: Already voted for candidate 3f397b9469554143884dc544bf64282a in this term.
I20240507 08:06:04.945650 29910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f"
I20240507 08:06:04.947060 29983 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 1 election: Requested vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:04.947026 29716 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5; no voters: 3f397b9469554143884dc544bf64282a, 64580d7ab4d84b86b6bb4a3e38fbd43f
I20240507 08:06:04.948412 29983 raft_consensus.cc:2726] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240507 08:06:04.942493 29911 raft_consensus.cc:2454] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f397b9469554143884dc544bf64282a in term 1.
I20240507 08:06:04.950139 29799 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f397b9469554143884dc544bf64282a, 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 3cebffdfceb24f75ae9450fce8382fa5
I20240507 08:06:04.952394 29965 raft_consensus.cc:2781] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:04.973851 29965 raft_consensus.cc:686] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 1 LEADER]: Becoming Leader. State: Replica: 3f397b9469554143884dc544bf64282a, State: Running, Role: LEADER
I20240507 08:06:04.974936 29965 consensus_queue.cc:237] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:04.989317 29983 consensus_queue.cc:1035] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:06:04.993083 29983 consensus_queue.cc:1035] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:06:05.000330 29965 consensus_queue.cc:1035] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:06:05.005162 29964 raft_consensus.cc:482] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:05.005677 29964 raft_consensus.cc:504] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:05.009312 29754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:05.009681 29833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:05.012462 29877 leader_election.cc:302] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 3cebffdfceb24f75ae9450fce8382fa5, 3f397b9469554143884dc544bf64282a
I20240507 08:06:05.013722 29964 leader_election.cc:288] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 3f397b9469554143884dc544bf64282a (127.27.100.130:36791)
I20240507 08:06:05.014346 29964 raft_consensus.cc:2726] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:06:05.019837 29910 raft_consensus.cc:3037] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:05.029487 29970 consensus_queue.cc:1035] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240507 08:06:05.065129 29877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.100.130:36791, user_credentials={real_user=slave}} blocked reactor thread for 52765.7us
I20240507 08:06:05.071157 29551 catalog_manager.cc:5484] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f397b9469554143884dc544bf64282a (127.27.100.130). New cstate: current_term: 1 leader_uuid: "3f397b9469554143884dc544bf64282a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:05.076401 29965 consensus_queue.cc:1035] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:06:05.089229 29965 consensus_queue.cc:1035] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:06:05.350569 29754 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed"
dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5"
 from {username='slave'} at 127.0.0.1:43820
I20240507 08:06:05.351256 29754 raft_consensus.cc:580] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 LEADER]: Received request to step down
I20240507 08:06:05.351545 29754 raft_consensus.cc:3032] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:06:05.351826 29754 raft_consensus.cc:729] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3cebffdfceb24f75ae9450fce8382fa5, State: Running, Role: LEADER
I20240507 08:06:05.352581 29754 consensus_queue.cc:260] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:05.353711 29754 raft_consensus.cc:3037] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:06:05.461992 30003 raft_consensus.cc:482] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3cebffdfceb24f75ae9450fce8382fa5)
I20240507 08:06:05.462623 30003 raft_consensus.cc:504] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:05.464604 30003 leader_election.cc:288] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455)
I20240507 08:06:05.465379 29831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:05.465559 29754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:05.466259 29754 raft_consensus.cc:2454] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64580d7ab4d84b86b6bb4a3e38fbd43f in term 2.
I20240507 08:06:05.467164 29876 leader_election.cc:302] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5, 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 3f397b9469554143884dc544bf64282a
I20240507 08:06:05.467922 30003 raft_consensus.cc:2781] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:06:05.468271 30003 raft_consensus.cc:482] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3cebffdfceb24f75ae9450fce8382fa5)
I20240507 08:06:05.468534 30003 raft_consensus.cc:3037] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:06:05.473253 30003 raft_consensus.cc:504] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:05.475184 30003 leader_election.cc:288] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 2 election: Requested vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455)
I20240507 08:06:05.475539 29831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a"
I20240507 08:06:05.475898 29754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5"
I20240507 08:06:05.480844 29754 raft_consensus.cc:2454] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64580d7ab4d84b86b6bb4a3e38fbd43f in term 2.
I20240507 08:06:05.481685 29876 leader_election.cc:302] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5, 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 3f397b9469554143884dc544bf64282a
I20240507 08:06:05.482394 30003 raft_consensus.cc:2781] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:06:05.482937 30003 raft_consensus.cc:686] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 2 LEADER]: Becoming Leader. State: Replica: 64580d7ab4d84b86b6bb4a3e38fbd43f, State: Running, Role: LEADER
I20240507 08:06:05.483819 30003 consensus_queue.cc:237] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:05.491312 29549 catalog_manager.cc:5484] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f reported cstate change: term changed from 1 to 2, leader changed from 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129) to 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131). New cstate: current_term: 2 leader_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:05.548741 29965 raft_consensus.cc:482] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3cebffdfceb24f75ae9450fce8382fa5)
I20240507 08:06:05.549201 29965 raft_consensus.cc:504] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:05.550773 29965 leader_election.cc:288] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631), 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455)
I20240507 08:06:05.551578 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:05.551779 29755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:05.552656 29755 raft_consensus.cc:2379] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f397b9469554143884dc544bf64282a in current term 2: Already voted for candidate 64580d7ab4d84b86b6bb4a3e38fbd43f in this term.
I20240507 08:06:05.553517 29798 leader_election.cc:302] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f397b9469554143884dc544bf64282a; no voters: 3cebffdfceb24f75ae9450fce8382fa5, 64580d7ab4d84b86b6bb4a3e38fbd43f
I20240507 08:06:05.554251 29965 raft_consensus.cc:3037] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:06:05.560945 29965 raft_consensus.cc:2726] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:06:05.571717 29831 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f"
dest_uuid: "3f397b9469554143884dc544bf64282a"
 from {username='slave'} at 127.0.0.1:45980
I20240507 08:06:05.572304 29831 raft_consensus.cc:580] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 1 LEADER]: Received request to step down
I20240507 08:06:05.572556 29831 raft_consensus.cc:3032] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:06:05.572790 29831 raft_consensus.cc:729] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f397b9469554143884dc544bf64282a, State: Running, Role: LEADER
I20240507 08:06:05.573331 29831 consensus_queue.cc:260] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:05.574180 29831 raft_consensus.cc:3037] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:06:05.574967 29833 raft_consensus.cc:1261] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Refusing update from remote peer 64580d7ab4d84b86b6bb4a3e38fbd43f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240507 08:06:05.576186 30003 consensus_queue.cc:1035] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:05.585623 29755 raft_consensus.cc:1261] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Refusing update from remote peer 64580d7ab4d84b86b6bb4a3e38fbd43f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240507 08:06:05.587409 29964 consensus_queue.cc:1035] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:05.698545 30028 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f397b9469554143884dc544bf64282a)
I20240507 08:06:05.699153 30028 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:05.701843 30028 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 3f397b9469554143884dc544bf64282a (127.27.100.130:36791)
I20240507 08:06:05.704355 29754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:05.704274 29831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:05.705780 29831 raft_consensus.cc:2454] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64580d7ab4d84b86b6bb4a3e38fbd43f in term 2.
I20240507 08:06:05.707616 29877 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f397b9469554143884dc544bf64282a, 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 3cebffdfceb24f75ae9450fce8382fa5
I20240507 08:06:05.712601 30029 raft_consensus.cc:2781] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:06:05.713085 30029 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f397b9469554143884dc544bf64282a)
I20240507 08:06:05.713542 30029 raft_consensus.cc:3037] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:06:05.719986 30029 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:05.721741 30029 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 2 election: Requested vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 3f397b9469554143884dc544bf64282a (127.27.100.130:36791)
I20240507 08:06:05.722528 29755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5"
I20240507 08:06:05.723204 29833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a"
I20240507 08:06:05.728101 29833 raft_consensus.cc:2454] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64580d7ab4d84b86b6bb4a3e38fbd43f in term 2.
I20240507 08:06:05.728906 29877 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f397b9469554143884dc544bf64282a, 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 3cebffdfceb24f75ae9450fce8382fa5
I20240507 08:06:05.729580 30029 raft_consensus.cc:2781] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:06:05.729997 30029 raft_consensus.cc:686] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 2 LEADER]: Becoming Leader. State: Replica: 64580d7ab4d84b86b6bb4a3e38fbd43f, State: Running, Role: LEADER
I20240507 08:06:05.730724 30029 consensus_queue.cc:237] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:05.737686 29550 catalog_manager.cc:5484] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f reported cstate change: term changed from 1 to 2, leader changed from 3f397b9469554143884dc544bf64282a (127.27.100.130) to 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131). New cstate: current_term: 2 leader_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:05.747301 30011 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f397b9469554143884dc544bf64282a)
I20240507 08:06:05.747905 30011 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:05.749935 30011 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:05.751518 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:05.753079 29833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:05.753883 29833 raft_consensus.cc:2379] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3cebffdfceb24f75ae9450fce8382fa5 in current term 2: Already voted for candidate 64580d7ab4d84b86b6bb4a3e38fbd43f in this term.
I20240507 08:06:05.754925 29716 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5; no voters: 3f397b9469554143884dc544bf64282a, 64580d7ab4d84b86b6bb4a3e38fbd43f
I20240507 08:06:05.755759 30011 raft_consensus.cc:3037] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:06:05.760053 30011 raft_consensus.cc:2726] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:06:05.780155 29833 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69"
dest_uuid: "3f397b9469554143884dc544bf64282a"
 from {username='slave'} at 127.0.0.1:45980
I20240507 08:06:05.781257 29833 raft_consensus.cc:580] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 1 LEADER]: Received request to step down
I20240507 08:06:05.783162 29833 raft_consensus.cc:3032] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:06:05.783567 29833 raft_consensus.cc:729] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f397b9469554143884dc544bf64282a, State: Running, Role: LEADER
I20240507 08:06:05.784541 29833 consensus_queue.cc:260] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:05.787426 29833 raft_consensus.cc:3037] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:06:05.827797 29833 raft_consensus.cc:1261] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Refusing update from remote peer 64580d7ab4d84b86b6bb4a3e38fbd43f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240507 08:06:05.828944 30028 consensus_queue.cc:1035] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:05.837306 29755 raft_consensus.cc:1261] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Refusing update from remote peer 64580d7ab4d84b86b6bb4a3e38fbd43f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240507 08:06:05.847468 30029 consensus_queue.cc:1035] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:05.932541 30011 raft_consensus.cc:482] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f397b9469554143884dc544bf64282a)
I20240507 08:06:05.933144 30011 raft_consensus.cc:504] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:05.936877 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:05.937726 29909 raft_consensus.cc:2454] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cebffdfceb24f75ae9450fce8382fa5 in term 1.
I20240507 08:06:05.958012 30011 leader_election.cc:288] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:05.958880 29833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:05.959643 29833 raft_consensus.cc:2454] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cebffdfceb24f75ae9450fce8382fa5 in term 2.
I20240507 08:06:06.149814 30011 raft_consensus.cc:482] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 64580d7ab4d84b86b6bb4a3e38fbd43f)
I20240507 08:06:06.150463 30011 raft_consensus.cc:504] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:06.152045 30011 leader_election.cc:288] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:06.157368 30030 raft_consensus.cc:482] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66b1fdcb58294d68ad651ac2fa49acb5)
I20240507 08:06:06.157940 30030 raft_consensus.cc:504] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:06.159413 30030 leader_election.cc:288] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005)
I20240507 08:06:06.168572 29497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f03eb788ff4a4820b31171b231de9d41" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" is_pre_election: true
I20240507 08:06:06.169327 29497 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03eb788ff4a4820b31171b231de9d41 in term 1.
I20240507 08:06:06.170578 29603 leader_election.cc:302] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b50ddd4dc5d4bcb84e234822fcc7f33, f03eb788ff4a4820b31171b231de9d41; no voters: 
I20240507 08:06:06.171401 30030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:06:06.171798 30030 raft_consensus.cc:482] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66b1fdcb58294d68ad651ac2fa49acb5)
I20240507 08:06:06.172173 30030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:06:06.198789 30030 raft_consensus.cc:504] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:06.200727 30030 leader_election.cc:288] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [CANDIDATE]: Term 2 election: Requested vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005)
I20240507 08:06:06.201826 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f03eb788ff4a4820b31171b231de9d41" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5"
I20240507 08:06:06.202913 29497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f03eb788ff4a4820b31171b231de9d41" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33"
I20240507 08:06:06.203627 29497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:06:06.216017 29497 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03eb788ff4a4820b31171b231de9d41 in term 2.
I20240507 08:06:06.217573 29603 leader_election.cc:302] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8b50ddd4dc5d4bcb84e234822fcc7f33, f03eb788ff4a4820b31171b231de9d41; no voters: 66b1fdcb58294d68ad651ac2fa49acb5
I20240507 08:06:06.219166 30030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:06:06.220417 30030 raft_consensus.cc:686] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 2 LEADER]: Becoming Leader. State: Replica: f03eb788ff4a4820b31171b231de9d41, State: Running, Role: LEADER
I20240507 08:06:06.221134 30030 consensus_queue.cc:237] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:06.615674 29561 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:06:06.616178 29561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 66b1fdcb58294d68ad651ac2fa49acb5, State: Running, Role: LEADER
I20240507 08:06:06.617165 29561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:06.618381 29561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:06:06.674561 30032 raft_consensus.cc:482] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:06.675434 30032 raft_consensus.cc:504] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:06.678269 30032 leader_election.cc:288] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:06.745563 29561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 2 FOLLOWER]: Refusing update from remote peer f03eb788ff4a4820b31171b231de9d41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240507 08:06:07.297025 29689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:07.297514 29689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
W20240507 08:06:07.211884 29776 debug-util.cc:398] Leaking SignalData structure 0x7b08000aed00 after lost signal to thread 28053
I20240507 08:06:07.272385 29755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:07.284608 30036 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 64580d7ab4d84b86b6bb4a3e38fbd43f)
I20240507 08:06:07.288962 29965 raft_consensus.cc:482] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 64580d7ab4d84b86b6bb4a3e38fbd43f)
W20240507 08:06:07.086222 29798 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.100.129:35455, user_credentials={real_user=slave}} blocked reactor thread for 166036us
I20240507 08:06:07.298696 29755 raft_consensus.cc:2454] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f397b9469554143884dc544bf64282a in term 1.
I20240507 08:06:07.298920 30036 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:07.300158 29965 raft_consensus.cc:504] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:07.300453 29755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:07.300940 30036 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:07.301241 29755 raft_consensus.cc:2454] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f397b9469554143884dc544bf64282a in term 1.
I20240507 08:06:07.302294 29615 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44030 (request call id 28) took 1106 ms. Trace:
I20240507 08:06:07.303259 29965 leader_election.cc:288] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631), 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455)
I20240507 08:06:07.304598 29754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:07.302498 29755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:07.348851 29615 rpcz_store.cc:276] 0507 08:06:06.195611 (+     0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:06.195726 (+   115us) service_pool.cc:225] Handling call
0507 08:06:07.301225 (+1105499us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b540015f254. stack: 00007f447ae7eba3 00007f447ae7eaa0 00007f447a7e3584 00000000003bd474 00000000003bd409 00007f44816a7d30 00007f448172b17a 00007f4483d41b1c 00007f4483e3ba8e 00007f4482149a44 00007f44821499d2 00007f4482149901 00007f448214987c 00007f4482148b32 00007f447f9e3bcc 00007f447f9e3006
0507 08:06:07.302093 (+   868us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8100224}
I20240507 08:06:07.302778 29798 leader_election.cc:302] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5, 3f397b9469554143884dc544bf64282a; no voters: 
W20240507 08:06:07.294736 29562 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36288 (request call id 2) took 1099 ms (client timeout 222 ms). Trace:
W20240507 08:06:07.298357 29601 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.013s	sys 0.026s
I20240507 08:06:07.317085 30011 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 64580d7ab4d84b86b6bb4a3e38fbd43f)
W20240507 08:06:07.441509 29601 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.797s	user 0.019s	sys 0.026s
I20240507 08:06:07.441905 30011 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:07.443997 30011 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
W20240507 08:06:07.351989 29798 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.100.129:35455, user_credentials={real_user=slave}} blocked reactor thread for 49152.2us
W20240507 08:06:07.351819 29562 rpcz_store.cc:269] 0507 08:06:06.195049 (+     0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:07.294339 (+1099290us) service_pool.cc:212] Skipping call since client already timed out
0507 08:06:07.294663 (+   324us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:06:07.360414 29497 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 2 FOLLOWER]: Refusing update from remote peer f03eb788ff4a4820b31171b231de9d41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240507 08:06:07.352192 29965 raft_consensus.cc:2781] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:07.366154 29626 raft_consensus.cc:1226] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 2 LEADER]: Rejecting Update request from peer 66b1fdcb58294d68ad651ac2fa49acb5 for earlier term 1. Current term is 2. Ops: []
I20240507 08:06:07.446372 29965 raft_consensus.cc:482] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:07.446802 29965 raft_consensus.cc:3037] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:07.394574 29908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:07.448568 30037 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240507 08:06:07.412001 29833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:07.394155 29911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:07.452682 30039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:07.453143 29799 leader_election.cc:302] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f397b9469554143884dc544bf64282a; no voters: 3cebffdfceb24f75ae9450fce8382fa5, 64580d7ab4d84b86b6bb4a3e38fbd43f
I20240507 08:06:07.453325 30039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:07.414053 30028 raft_consensus.cc:482] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f397b9469554143884dc544bf64282a)
I20240507 08:06:07.454759 30036 raft_consensus.cc:2726] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:07.454798 30028 raft_consensus.cc:504] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:07.456073 29496 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 2 FOLLOWER]: Rejecting Update request from peer 66b1fdcb58294d68ad651ac2fa49acb5 for earlier term 1. Current term is 2. Ops: []
I20240507 08:06:07.457242 30028 leader_election.cc:288] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 3f397b9469554143884dc544bf64282a (127.27.100.130:36791)
I20240507 08:06:07.457350 30039 leader_election.cc:288] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:07.417093 29907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:07.458158 29907 raft_consensus.cc:2454] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cebffdfceb24f75ae9450fce8382fa5 in term 1.
I20240507 08:06:07.420991 29910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:07.459096 29832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:07.421665 29698 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240507 08:06:07.461027 29716 leader_election.cc:302] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5, 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 
I20240507 08:06:07.434423 29906 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed"
dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f"
 from {username='slave'} at 127.0.0.1:39970
I20240507 08:06:07.461971 29906 raft_consensus.cc:580] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 2 LEADER]: Received request to step down
I20240507 08:06:07.462328 29906 raft_consensus.cc:3032] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:06:07.436301 30031 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f03eb788ff4a4820b31171b231de9d41. Latest consensus state: current_term: 2 leader_uuid: "f03eb788ff4a4820b31171b231de9d41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:07.462455 29716 leader_election.cc:302] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5; no voters: 3f397b9469554143884dc544bf64282a, 64580d7ab4d84b86b6bb4a3e38fbd43f
I20240507 08:06:07.462857 30031 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:07.439112 29689 leader_election.cc:288] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:07.440521 29495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" is_pre_election: true
I20240507 08:06:07.440830 29627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f03eb788ff4a4820b31171b231de9d41" is_pre_election: true
I20240507 08:06:07.464036 29906 raft_consensus.cc:729] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64580d7ab4d84b86b6bb4a3e38fbd43f, State: Running, Role: LEADER
I20240507 08:06:07.464421 29799 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f397b9469554143884dc544bf64282a; no voters: 3cebffdfceb24f75ae9450fce8382fa5, 64580d7ab4d84b86b6bb4a3e38fbd43f
I20240507 08:06:07.464037 29495 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b1fdcb58294d68ad651ac2fa49acb5 in term 2.
I20240507 08:06:07.465284 29562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" is_pre_election: true
I20240507 08:06:07.465121 29906 consensus_queue.cc:260] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:07.464062 29911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
W20240507 08:06:07.441998 29601 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.100.189:36005, user_credentials={real_user=slave}} blocked reactor thread for 797976us
I20240507 08:06:07.466583 29906 raft_consensus.cc:3037] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:07.467170 29833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:07.467746 30011 raft_consensus.cc:2781] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240507 08:06:07.433292 29776 debug-util.cc:398] Leaking SignalData structure 0x7b0800140880 after lost signal to thread 29521
I20240507 08:06:07.468415 30011 raft_consensus.cc:482] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f397b9469554143884dc544bf64282a)
I20240507 08:06:07.468888 30011 raft_consensus.cc:3037] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:06:07.415113 29831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:07.471444 30037 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240507 08:06:07.471973 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f03eb788ff4a4820b31171b231de9d41" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" is_pre_election: true
I20240507 08:06:07.473773 29626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f03eb788ff4a4820b31171b231de9d41" is_pre_election: true
W20240507 08:06:07.469547 29776 debug-util.cc:398] Leaking SignalData structure 0x7b08000fd8a0 after lost signal to thread 29586
I20240507 08:06:07.467244 30036 raft_consensus.cc:2726] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240507 08:06:07.479683 29776 debug-util.cc:398] Leaking SignalData structure 0x7b080022e0c0 after lost signal to thread 29651
W20240507 08:06:07.481470 29776 debug-util.cc:398] Leaking SignalData structure 0x7b0800102d60 after lost signal to thread 29779
W20240507 08:06:07.481765 29776 debug-util.cc:398] Leaking SignalData structure 0x7b0800102d80 after lost signal to thread 29780
W20240507 08:06:07.482232 29776 debug-util.cc:398] Leaking SignalData structure 0x7b0800037780 after lost signal to thread 29782
I20240507 08:06:07.481732 29965 raft_consensus.cc:504] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:07.483157 29832 raft_consensus.cc:2354] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 64580d7ab4d84b86b6bb4a3e38fbd43f for earlier term 2. Current term is 3.
I20240507 08:06:07.412439 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:07.483981 29909 raft_consensus.cc:2454] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f397b9469554143884dc544bf64282a in term 1.
I20240507 08:06:07.485539 29538 leader_election.cc:302] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b1fdcb58294d68ad651ac2fa49acb5, 8b50ddd4dc5d4bcb84e234822fcc7f33; no voters: 
I20240507 08:06:07.486068 29877 leader_election.cc:398] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3f397b9469554143884dc544bf64282a (127.27.100.130:36791) with higher term. Message: Invalid argument: T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 64580d7ab4d84b86b6bb4a3e38fbd43f for earlier term 2. Current term is 3.
I20240507 08:06:07.486395 29877 leader_election.cc:401] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:06:07.488410 30029 raft_consensus.cc:3037] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 1 FOLLOWER]: Advancing to term 3
I20240507 08:06:07.466207 29562 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b50ddd4dc5d4bcb84e234822fcc7f33 in term 2.
I20240507 08:06:07.492307 29471 leader_election.cc:302] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b1fdcb58294d68ad651ac2fa49acb5, 8b50ddd4dc5d4bcb84e234822fcc7f33; no voters: f03eb788ff4a4820b31171b231de9d41
I20240507 08:06:07.493259 30039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:07.493726 30039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:07.494092 30039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:07.496165 29689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:07.498561 29689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:07.498932 29689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:07.500901 29755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:07.502072 30039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:07.504062 29616 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44020 (request call id 33) took 1254 ms. Trace:
I20240507 08:06:07.504462 29616 rpcz_store.cc:276] 0507 08:06:06.196544 (+     0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:07.366578 (+1170034us) service_pool.cc:225] Handling call
0507 08:06:07.450740 (+ 84162us) spinlock_profiling.cc:243] Waited 81.6 ms on lock 0x7b540015f254. stack: 00007f447ae7eba3 00007f447ae7eaa0 00007f447a7e3584 00000000003bd474 00000000003bd409 00007f44816a7d30 00007f448172b17a 00007f4483d41b1c 00007f4483e3ba8e 00007f4482149a44 00007f44821499d2 00007f4482149901 00007f448214987c 00007f4482148b32 00007f447f9e3bcc 00007f447f9e3006
0507 08:06:07.451301 (+   561us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":187568384}
I20240507 08:06:07.507390 29562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5"
I20240507 08:06:07.509003 30029 raft_consensus.cc:2726] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 3f397b9469554143884dc544bf64282a (127.27.100.130:36791) with higher term. Message: Invalid argument: T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 64580d7ab4d84b86b6bb4a3e38fbd43f for earlier term 2. Current term is 3.
I20240507 08:06:07.510754 29626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f03eb788ff4a4820b31171b231de9d41"
I20240507 08:06:07.511461 29496 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 3 FOLLOWER]: Rejecting Update request from peer f03eb788ff4a4820b31171b231de9d41 for earlier term 2. Current term is 3. Ops: [2.12-2.12]
I20240507 08:06:07.512668 29474 leader_election.cc:302] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b50ddd4dc5d4bcb84e234822fcc7f33; no voters: 66b1fdcb58294d68ad651ac2fa49acb5, f03eb788ff4a4820b31171b231de9d41
I20240507 08:06:07.512681 29754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5"
I20240507 08:06:07.512950 30037 consensus_queue.cc:1046] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 }, Status: INVALID_TERM, Last received: 1.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240507 08:06:07.517311 29965 leader_election.cc:288] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 3 election: Requested vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:07.518855 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f"
I20240507 08:06:07.520273 30039 leader_election.cc:288] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 3 election: Requested vote from peers 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:07.521180 30039 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240507 08:06:07.483340 29776 debug-util.cc:398] Leaking SignalData structure 0x7b08000ff820 after lost signal to thread 29857
I20240507 08:06:07.527837 29909 raft_consensus.cc:2454] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f397b9469554143884dc544bf64282a in term 3.
I20240507 08:06:07.509121 29689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:07.529021 29799 leader_election.cc:302] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f397b9469554143884dc544bf64282a, 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 3cebffdfceb24f75ae9450fce8382fa5
I20240507 08:06:07.529487 29561 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 3 FOLLOWER]: Rejecting Update request from peer f03eb788ff4a4820b31171b231de9d41 for earlier term 2. Current term is 3. Ops: [2.12-2.12]
I20240507 08:06:07.529909 30036 raft_consensus.cc:2781] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:06:07.530473 30036 raft_consensus.cc:686] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 3 LEADER]: Becoming Leader. State: Replica: 3f397b9469554143884dc544bf64282a, State: Running, Role: LEADER
I20240507 08:06:07.531455 30036 consensus_queue.cc:237] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:07.532873 30037 consensus_queue.cc:1046] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 }, Status: INVALID_TERM, Last received: 1.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240507 08:06:07.535524 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33"
I20240507 08:06:07.536448 29496 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b1fdcb58294d68ad651ac2fa49acb5 in current term 3: Already voted for candidate 8b50ddd4dc5d4bcb84e234822fcc7f33 in this term.
W20240507 08:06:07.538090 29716 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.100.130:36791, user_credentials={real_user=slave}} blocked reactor thread for 75744.1us
I20240507 08:06:07.539399 29716 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5; no voters: 3f397b9469554143884dc544bf64282a, 64580d7ab4d84b86b6bb4a3e38fbd43f
I20240507 08:06:07.519055 30011 raft_consensus.cc:504] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:07.519466 30031 raft_consensus.cc:3032] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:06:07.546371 29626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f03eb788ff4a4820b31171b231de9d41"
I20240507 08:06:07.544327 30031 raft_consensus.cc:729] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f03eb788ff4a4820b31171b231de9d41, State: Running, Role: LEADER
I20240507 08:06:07.548451 29539 leader_election.cc:302] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b1fdcb58294d68ad651ac2fa49acb5; no voters: 8b50ddd4dc5d4bcb84e234822fcc7f33, f03eb788ff4a4820b31171b231de9d41
I20240507 08:06:07.548307 30031 consensus_queue.cc:260] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:07.544423 29689 leader_election.cc:288] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 3 election: Requested vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:07.549738 30009 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:06:07.550987 30031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:07.553555 30044 raft_consensus.cc:2726] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:07.554576 30044 raft_consensus.cc:2726] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:07.558080 29755 raft_consensus.cc:2379] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 64580d7ab4d84b86b6bb4a3e38fbd43f in current term 2: Already voted for candidate 3cebffdfceb24f75ae9450fce8382fa5 in this term.
W20240507 08:06:07.565075 30045 catalog_manager.cc:1409] Failed waiting for node to catch up after master election: Illegal state: Node f03eb788ff4a4820b31171b231de9d41 not leader. Raft Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:07.566552 30011 leader_election.cc:288] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 2 election: Requested vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:07.567634 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f"
I20240507 08:06:07.567787 29832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e5e59c50f54e548e29ec71cabf4e69" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a"
I20240507 08:06:07.568396 29909 raft_consensus.cc:2354] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3cebffdfceb24f75ae9450fce8382fa5 for earlier term 2. Current term is 3.
I20240507 08:06:07.569958 29716 leader_election.cc:398] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 2 election: Vote denied by peer 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631) with higher term. Message: Invalid argument: T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3cebffdfceb24f75ae9450fce8382fa5 for earlier term 2. Current term is 3.
I20240507 08:06:07.570354 29716 leader_election.cc:401] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240507 08:06:07.571028 30011 raft_consensus.cc:3037] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:07.578264 30011 raft_consensus.cc:2726] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631) with higher term. Message: Invalid argument: T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3cebffdfceb24f75ae9450fce8382fa5 for earlier term 2. Current term is 3.
I20240507 08:06:07.629606 29909 raft_consensus.cc:1261] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Refusing update from remote peer 3f397b9469554143884dc544bf64282a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:06:07.630260 29755 raft_consensus.cc:1261] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Refusing update from remote peer 3f397b9469554143884dc544bf64282a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:06:07.631603 29965 consensus_queue.cc:1035] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:07.633231 30032 consensus_queue.cc:1035] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:06:07.708441 30036 raft_consensus.cc:482] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 64580d7ab4d84b86b6bb4a3e38fbd43f)
I20240507 08:06:07.709077 30036 raft_consensus.cc:504] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:07.714313 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:07.714979 29909 raft_consensus.cc:2454] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f397b9469554143884dc544bf64282a in term 3.
I20240507 08:06:07.716651 29799 leader_election.cc:302] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f397b9469554143884dc544bf64282a, 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 
I20240507 08:06:07.717561 30032 raft_consensus.cc:2781] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:07.717921 30032 raft_consensus.cc:482] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 64580d7ab4d84b86b6bb4a3e38fbd43f)
I20240507 08:06:07.718279 30032 raft_consensus.cc:3037] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:07.745355 29754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:07.746258 29754 raft_consensus.cc:2454] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f397b9469554143884dc544bf64282a in term 2.
I20240507 08:06:07.748553 30036 leader_election.cc:288] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631), 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455)
I20240507 08:06:07.750711 30009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:07.751346 30009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:07.756191 29626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" candidate_term: 4 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f03eb788ff4a4820b31171b231de9d41" is_pre_election: true
I20240507 08:06:07.742233 30032 raft_consensus.cc:504] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:07.757110 29626 raft_consensus.cc:2396] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66b1fdcb58294d68ad651ac2fa49acb5 for term 4 because replica has last-logged OpId of term: 2 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 11.
I20240507 08:06:07.758984 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f"
I20240507 08:06:07.764067 30032 leader_election.cc:288] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 3 election: Requested vote from peers 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631), 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455)
I20240507 08:06:07.764117 30046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:07.764780 29754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffdf4714a4ad28391f055890b46ed" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5"
I20240507 08:06:07.764796 30046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:07.765939 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" candidate_term: 4 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" is_pre_election: true
I20240507 08:06:07.766796 29496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b1fdcb58294d68ad651ac2fa49acb5 in term 3.
I20240507 08:06:07.767355 30046 leader_election.cc:288] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:07.763833 30009 leader_election.cc:288] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:07.765540 29754 raft_consensus.cc:3037] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:07.769791 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 4 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" is_pre_election: true
I20240507 08:06:07.770431 29561 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b50ddd4dc5d4bcb84e234822fcc7f33 in term 3.
I20240507 08:06:07.770606 30031 raft_consensus.cc:482] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:07.771068 30031 raft_consensus.cc:504] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:07.771790 29471 leader_election.cc:302] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b1fdcb58294d68ad651ac2fa49acb5, 8b50ddd4dc5d4bcb84e234822fcc7f33; no voters: 
I20240507 08:06:07.773092 30031 leader_election.cc:288] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005)
I20240507 08:06:07.773416 29538 leader_election.cc:302] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66b1fdcb58294d68ad651ac2fa49acb5, 8b50ddd4dc5d4bcb84e234822fcc7f33; no voters: f03eb788ff4a4820b31171b231de9d41
I20240507 08:06:07.775141 29626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 4 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f03eb788ff4a4820b31171b231de9d41" is_pre_election: true
I20240507 08:06:07.776058 29626 raft_consensus.cc:2396] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8b50ddd4dc5d4bcb84e234822fcc7f33 for term 4 because replica has last-logged OpId of term: 2 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 11.
I20240507 08:06:07.779244 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f03eb788ff4a4820b31171b231de9d41" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" is_pre_election: true
I20240507 08:06:07.779577 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f03eb788ff4a4820b31171b231de9d41" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" is_pre_election: true
I20240507 08:06:07.779922 29496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03eb788ff4a4820b31171b231de9d41 in term 3.
I20240507 08:06:07.780140 29561 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03eb788ff4a4820b31171b231de9d41 in term 3.
I20240507 08:06:07.780411 29909 raft_consensus.cc:2454] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f397b9469554143884dc544bf64282a in term 3.
I20240507 08:06:07.781067 29603 leader_election.cc:302] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b50ddd4dc5d4bcb84e234822fcc7f33, f03eb788ff4a4820b31171b231de9d41; no voters: 
I20240507 08:06:07.781721 29799 leader_election.cc:302] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f397b9469554143884dc544bf64282a, 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 
I20240507 08:06:07.782285 30031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:07.782667 30031 raft_consensus.cc:482] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:07.782991 30031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:07.784884 30046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:07.785816 30046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:07.786250 30046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:07.783299 30032 raft_consensus.cc:2781] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:06:07.785440 30009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:07.790858 30032 raft_consensus.cc:686] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 3 LEADER]: Becoming Leader. State: Replica: 3f397b9469554143884dc544bf64282a, State: Running, Role: LEADER
I20240507 08:06:07.791394 30009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:07.792630 30009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:07.793087 30032 consensus_queue.cc:237] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } }
I20240507 08:06:07.799178 30031 raft_consensus.cc:504] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:07.775163 29754 raft_consensus.cc:2454] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f397b9469554143884dc544bf64282a in term 3.
I20240507 08:06:07.812045 30046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:07.815626 30031 leader_election.cc:288] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [CANDIDATE]: Term 4 election: Requested vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005)
I20240507 08:06:07.824193 29562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f03eb788ff4a4820b31171b231de9d41" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5"
I20240507 08:06:07.824354 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 4 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5"
I20240507 08:06:07.815528 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f03eb788ff4a4820b31171b231de9d41" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33"
I20240507 08:06:07.829632 29496 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f03eb788ff4a4820b31171b231de9d41 in current term 4: Already voted for candidate 8b50ddd4dc5d4bcb84e234822fcc7f33 in this term.
I20240507 08:06:07.829897 29626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 4 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f03eb788ff4a4820b31171b231de9d41"
I20240507 08:06:07.830777 29626 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8b50ddd4dc5d4bcb84e234822fcc7f33 in current term 4: Already voted for candidate f03eb788ff4a4820b31171b231de9d41 in this term.
I20240507 08:06:07.832286 29474 leader_election.cc:302] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b50ddd4dc5d4bcb84e234822fcc7f33; no voters: 66b1fdcb58294d68ad651ac2fa49acb5, f03eb788ff4a4820b31171b231de9d41
I20240507 08:06:07.833348 30039 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:07.831941 30046 leader_election.cc:288] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 4 election: Requested vote from peers 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:07.838846 30009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:07.841782 30009 leader_election.cc:288] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 4 election: Requested vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:07.843909 29562 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f03eb788ff4a4820b31171b231de9d41 in current term 4: Already voted for candidate 66b1fdcb58294d68ad651ac2fa49acb5 in this term.
I20240507 08:06:07.845733 29601 leader_election.cc:302] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f03eb788ff4a4820b31171b231de9d41; no voters: 66b1fdcb58294d68ad651ac2fa49acb5, 8b50ddd4dc5d4bcb84e234822fcc7f33
I20240507 08:06:07.846817 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" candidate_term: 4 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33"
I20240507 08:06:07.847879 29496 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b1fdcb58294d68ad651ac2fa49acb5 in current term 4: Already voted for candidate 8b50ddd4dc5d4bcb84e234822fcc7f33 in this term.
I20240507 08:06:07.846973 30031 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:07.849238 29626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" candidate_term: 4 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f03eb788ff4a4820b31171b231de9d41"
I20240507 08:06:07.850119 29626 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 66b1fdcb58294d68ad651ac2fa49acb5 in current term 4: Already voted for candidate f03eb788ff4a4820b31171b231de9d41 in this term.
I20240507 08:06:07.851063 29539 leader_election.cc:302] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66b1fdcb58294d68ad651ac2fa49acb5; no voters: 8b50ddd4dc5d4bcb84e234822fcc7f33, f03eb788ff4a4820b31171b231de9d41
I20240507 08:06:07.851732 30009 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:07.884455 29909 raft_consensus.cc:1261] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Refusing update from remote peer 3f397b9469554143884dc544bf64282a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240507 08:06:07.885639 30036 consensus_queue.cc:1035] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:07.900071 29755 raft_consensus.cc:1261] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Refusing update from remote peer 3f397b9469554143884dc544bf64282a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240507 08:06:07.904031 30036 consensus_queue.cc:1035] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:08.065487 30046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:08.066115 30046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:08.068293 30046 leader_election.cc:288] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:08.069075 29562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 5 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" is_pre_election: true
I20240507 08:06:08.069275 29626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 5 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f03eb788ff4a4820b31171b231de9d41" is_pre_election: true
I20240507 08:06:08.069819 29562 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b50ddd4dc5d4bcb84e234822fcc7f33 in term 4.
I20240507 08:06:08.070088 29626 raft_consensus.cc:2396] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8b50ddd4dc5d4bcb84e234822fcc7f33 for term 5 because replica has last-logged OpId of term: 2 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 11.
I20240507 08:06:08.070827 29471 leader_election.cc:302] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b1fdcb58294d68ad651ac2fa49acb5, 8b50ddd4dc5d4bcb84e234822fcc7f33; no voters: 
I20240507 08:06:08.071563 30046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:08.071923 30046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:08.072207 30046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:08.076822 30046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:08.078632 30046 leader_election.cc:288] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 5 election: Requested vote from peers 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:08.079185 29626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 5 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f03eb788ff4a4820b31171b231de9d41"
I20240507 08:06:08.079028 29562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 5 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5"
I20240507 08:06:08.079787 29626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:08.079901 29562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:08.085196 29626 raft_consensus.cc:2396] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8b50ddd4dc5d4bcb84e234822fcc7f33 for term 5 because replica has last-logged OpId of term: 2 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 11.
I20240507 08:06:08.085559 30031 raft_consensus.cc:482] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:08.086011 30031 raft_consensus.cc:504] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:08.087235 29562 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b50ddd4dc5d4bcb84e234822fcc7f33 in term 5.
I20240507 08:06:08.087819 30031 leader_election.cc:288] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005)
I20240507 08:06:08.088542 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f03eb788ff4a4820b31171b231de9d41" candidate_term: 6 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" is_pre_election: true
I20240507 08:06:08.088950 29471 leader_election.cc:302] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b1fdcb58294d68ad651ac2fa49acb5, 8b50ddd4dc5d4bcb84e234822fcc7f33; no voters: 
I20240507 08:06:08.089215 29496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03eb788ff4a4820b31171b231de9d41 in term 5.
I20240507 08:06:08.089180 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f03eb788ff4a4820b31171b231de9d41" candidate_term: 6 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" is_pre_election: true
I20240507 08:06:08.090804 29561 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03eb788ff4a4820b31171b231de9d41 in term 5.
I20240507 08:06:08.090689 29603 leader_election.cc:302] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b50ddd4dc5d4bcb84e234822fcc7f33, f03eb788ff4a4820b31171b231de9d41; no voters: 
I20240507 08:06:08.092003 30031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:08.092698 30031 raft_consensus.cc:482] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:08.093055 30031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:08.094138 30046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:08.094939 30046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 5 LEADER]: Becoming Leader. State: Replica: 8b50ddd4dc5d4bcb84e234822fcc7f33, State: Running, Role: LEADER
I20240507 08:06:08.095927 30046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:08.100800 30039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b50ddd4dc5d4bcb84e234822fcc7f33. Latest consensus state: current_term: 5 leader_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:08.100805 30031 raft_consensus.cc:504] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:08.101774 30039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:08.103386 30059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:06:08.105741 30031 leader_election.cc:288] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [CANDIDATE]: Term 6 election: Requested vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005)
I20240507 08:06:08.107089 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f03eb788ff4a4820b31171b231de9d41" candidate_term: 6 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5"
I20240507 08:06:08.107765 29561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:08.110893 30059 catalog_manager.cc:669] Loaded metadata for table test-table [id=1d79b5e45252475d87377636deec0dd9]
I20240507 08:06:08.116763 29561 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03eb788ff4a4820b31171b231de9d41 in term 6.
I20240507 08:06:08.118070 30059 tablet_loader.cc:96] loaded metadata for tablet 55e5e59c50f54e548e29ec71cabf4e69 (table test-table [id=1d79b5e45252475d87377636deec0dd9])
I20240507 08:06:08.119608 30059 tablet_loader.cc:96] loaded metadata for tablet 6eeffdf4714a4ad28391f055890b46ed (table test-table [id=1d79b5e45252475d87377636deec0dd9])
I20240507 08:06:08.120348 29601 leader_election.cc:302] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b1fdcb58294d68ad651ac2fa49acb5, f03eb788ff4a4820b31171b231de9d41; no voters: 
I20240507 08:06:08.121011 30059 tablet_loader.cc:96] loaded metadata for tablet 71c9b00c40c6458f9a0fe4a3a57fc99f (table test-table [id=1d79b5e45252475d87377636deec0dd9])
I20240507 08:06:08.122440 30059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:06:08.124987 30031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:08.126884 30059 catalog_manager.cc:1259] Loaded cluster ID: a5e9cfaea5f8455aa6e905b407ec60c8
I20240507 08:06:08.127190 30059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:06:08.130028 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f03eb788ff4a4820b31171b231de9d41" candidate_term: 6 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33"
I20240507 08:06:08.132578 30031 raft_consensus.cc:686] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 6 LEADER]: Becoming Leader. State: Replica: f03eb788ff4a4820b31171b231de9d41, State: Running, Role: LEADER
I20240507 08:06:08.133525 30031 consensus_queue.cc:237] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.12, Last appended by leader: 12, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:08.143796 30059 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:06:08.147543 30060 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f03eb788ff4a4820b31171b231de9d41. Latest consensus state: current_term: 6 leader_uuid: "f03eb788ff4a4820b31171b231de9d41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:08.148425 30060 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:08.150418 30062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:06:08.151062 30059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33: Loaded TSK: 0
I20240507 08:06:08.152506 30059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:06:08.158164 30062 catalog_manager.cc:669] Loaded metadata for table test-table [id=1d79b5e45252475d87377636deec0dd9]
I20240507 08:06:08.162295 30062 tablet_loader.cc:96] loaded metadata for tablet 55e5e59c50f54e548e29ec71cabf4e69 (table test-table [id=1d79b5e45252475d87377636deec0dd9])
I20240507 08:06:08.163409 30062 tablet_loader.cc:96] loaded metadata for tablet 6eeffdf4714a4ad28391f055890b46ed (table test-table [id=1d79b5e45252475d87377636deec0dd9])
I20240507 08:06:08.164448 30062 tablet_loader.cc:96] loaded metadata for tablet 71c9b00c40c6458f9a0fe4a3a57fc99f (table test-table [id=1d79b5e45252475d87377636deec0dd9])
I20240507 08:06:08.165307 30062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:06:08.168483 30062 catalog_manager.cc:1259] Loaded cluster ID: a5e9cfaea5f8455aa6e905b407ec60c8
I20240507 08:06:08.168697 30062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:06:08.172060 30062 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:06:08.175338 30062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: Loaded TSK: 0
I20240507 08:06:08.176291 30062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:06:08.192957 29626 raft_consensus.cc:1226] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 6 LEADER]: Rejecting Update request from peer 8b50ddd4dc5d4bcb84e234822fcc7f33 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:08.194587 30046 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240507 08:06:08.195396 30046 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [LEADER]: Peer f03eb788ff4a4820b31171b231de9d41 log is divergent from this leader: its last log entry 6.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20240507 08:06:08.195935 30039 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:08.196317 30039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8b50ddd4dc5d4bcb84e234822fcc7f33, State: Running, Role: LEADER
I20240507 08:06:08.197001 30039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 5.12, Last appended by leader: 12, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:08.198047 30039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:08.219261 29496 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 6 FOLLOWER]: Refusing update from remote peer f03eb788ff4a4820b31171b231de9d41: Log matching property violated. Preceding OpId in replica: term: 5 index: 12. Preceding OpId from leader: term: 6 index: 13. (index mismatch)
I20240507 08:06:08.220593 30060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240507 08:06:08.221195 30060 consensus_queue.cc:1225] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [LEADER]: Peer 8b50ddd4dc5d4bcb84e234822fcc7f33 log is divergent from this leader: its last log entry 5.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20240507 08:06:08.223227 29496 pending_rounds.cc:76] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33: Aborting all ops after (but not including) 11
I20240507 08:06:08.223573 29496 pending_rounds.cc:97] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33: Aborting uncommitted NO_OP operation due to leader change: 5.12
I20240507 08:06:08.223829 29496 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:08.225346 30039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f03eb788ff4a4820b31171b231de9d41. Latest consensus state: current_term: 6 leader_uuid: "f03eb788ff4a4820b31171b231de9d41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:08.226131 30039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:08.232261 30046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "f03eb788ff4a4820b31171b231de9d41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:08.232985 30046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:08.233496 30046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "f03eb788ff4a4820b31171b231de9d41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:08.234198 30046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:08.238665 29561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 6 FOLLOWER]: Refusing update from remote peer f03eb788ff4a4820b31171b231de9d41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 6 index: 13. (index mismatch)
I20240507 08:06:08.242446 30065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20240507 08:06:08.249619 30031 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "f03eb788ff4a4820b31171b231de9d41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:08.250640 30031 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:08.264580 30069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "f03eb788ff4a4820b31171b231de9d41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:08.265178 30009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f03eb788ff4a4820b31171b231de9d41. Latest consensus state: current_term: 6 leader_uuid: "f03eb788ff4a4820b31171b231de9d41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:08.265712 30069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:08.266093 30009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:08.266579 30068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "f03eb788ff4a4820b31171b231de9d41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:08.267287 30068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:08.346158 29936 heartbeater.cc:503] Master 127.27.100.188:35355 was elected leader, sending a full tablet report...
I20240507 08:06:08.347867 30065 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "f03eb788ff4a4820b31171b231de9d41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:08.348778 30065 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:08.355115 29780 heartbeater.cc:503] Master 127.27.100.188:35355 was elected leader, sending a full tablet report...
I20240507 08:06:08.355779 29858 heartbeater.cc:503] Master 127.27.100.188:35355 was elected leader, sending a full tablet report...
I20240507 08:06:08.360702 29616 catalog_manager.cc:5484] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f reported cstate change: term changed from 2 to 3, leader changed from 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131) to 3f397b9469554143884dc544bf64282a (127.27.100.130). New cstate: current_term: 3 leader_uuid: "3f397b9469554143884dc544bf64282a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } }
I20240507 08:06:08.362521 29616 catalog_manager.cc:5484] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3f397b9469554143884dc544bf64282a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } } }
I20240507 08:06:08.466372 29909 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f"
dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f"
 from {username='slave'} at 127.0.0.1:39970
I20240507 08:06:08.466894 29909 raft_consensus.cc:580] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 2 LEADER]: Received request to step down
I20240507 08:06:08.467204 29909 raft_consensus.cc:3032] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:06:08.467514 29909 raft_consensus.cc:729] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64580d7ab4d84b86b6bb4a3e38fbd43f, State: Running, Role: LEADER
I20240507 08:06:08.468122 29909 consensus_queue.cc:260] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:08.469108 29909 raft_consensus.cc:3037] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:08.602517 30073 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 64580d7ab4d84b86b6bb4a3e38fbd43f)
I20240507 08:06:08.603001 30073 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:08.604804 30073 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:08.606020 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:08.606020 29832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a" is_pre_election: true
I20240507 08:06:08.606892 29909 raft_consensus.cc:2454] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cebffdfceb24f75ae9450fce8382fa5 in term 3.
I20240507 08:06:08.608448 29716 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5, 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 
I20240507 08:06:08.610461 30073 raft_consensus.cc:2781] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:08.610882 30073 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 64580d7ab4d84b86b6bb4a3e38fbd43f)
I20240507 08:06:08.611305 30073 raft_consensus.cc:3037] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:08.619900 30073 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:08.622021 30073 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 3 election: Requested vote from peers 3f397b9469554143884dc544bf64282a (127.27.100.130:36791), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:08.622715 29832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3f397b9469554143884dc544bf64282a"
I20240507 08:06:08.622915 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3cebffdfceb24f75ae9450fce8382fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f"
I20240507 08:06:08.627601 29909 raft_consensus.cc:2454] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cebffdfceb24f75ae9450fce8382fa5 in term 3.
I20240507 08:06:08.628443 29716 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cebffdfceb24f75ae9450fce8382fa5, 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 3f397b9469554143884dc544bf64282a
I20240507 08:06:08.629103 30073 raft_consensus.cc:2781] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:06:08.630154 30073 raft_consensus.cc:686] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 LEADER]: Becoming Leader. State: Replica: 3cebffdfceb24f75ae9450fce8382fa5, State: Running, Role: LEADER
I20240507 08:06:08.631331 30073 consensus_queue.cc:237] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:08.642091 29616 catalog_manager.cc:5484] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 reported cstate change: term changed from 2 to 3, leader changed from 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131) to 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129). New cstate: current_term: 3 leader_uuid: "3cebffdfceb24f75ae9450fce8382fa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:08.665673 30036 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 64580d7ab4d84b86b6bb4a3e38fbd43f)
I20240507 08:06:08.666214 30036 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:08.668172 30036 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:08.669314 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" is_pre_election: true
I20240507 08:06:08.670133 29909 raft_consensus.cc:2379] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f397b9469554143884dc544bf64282a in current term 3: Already voted for candidate 3cebffdfceb24f75ae9450fce8382fa5 in this term.
I20240507 08:06:08.673074 29754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c9b00c40c6458f9a0fe4a3a57fc99f" candidate_uuid: "3f397b9469554143884dc544bf64282a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3cebffdfceb24f75ae9450fce8382fa5" is_pre_election: true
I20240507 08:06:08.679260 29798 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f397b9469554143884dc544bf64282a; no voters: 3cebffdfceb24f75ae9450fce8382fa5, 64580d7ab4d84b86b6bb4a3e38fbd43f
I20240507 08:06:08.680388 30036 raft_consensus.cc:3037] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:08.694217 30036 raft_consensus.cc:2726] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:08.713111 29616 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44048:
table { table_id: "1d79b5e45252475d87377636deec0dd9" table_name: "test-table" } new_table_name: "NewTableName"
I20240507 08:06:08.750125 28050 master.cc:537] Master@127.27.100.188:35355 shutting down...
I20240507 08:06:08.760577 29909 raft_consensus.cc:1261] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Refusing update from remote peer 3cebffdfceb24f75ae9450fce8382fa5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240507 08:06:08.760505 29832 raft_consensus.cc:1261] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Refusing update from remote peer 3cebffdfceb24f75ae9450fce8382fa5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240507 08:06:08.762993 30074 consensus_queue.cc:1035] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:08.764429 30074 consensus_queue.cc:1035] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:08.898012 30089 tablet.cc:1722] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20240507 08:06:08.902241 30091 tablet.cc:1722] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20240507 08:06:08.912143 30095 tablet.cc:1722] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20240507 08:06:08.940802 30098 tablet.cc:1722] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20240507 08:06:08.944444 30093 tablet.cc:1722] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20240507 08:06:08.956015 30100 tablet.cc:1722] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20240507 08:06:08.958695 30097 tablet.cc:1722] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20240507 08:06:08.985323 30101 tablet.cc:1722] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20240507 08:06:09.008394 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 6 LEADER]: Raft consensus shutting down.
I20240507 08:06:09.009052 30095 tablet.cc:1722] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f: Alter schema from (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    PRIMARY KEY (key)
) version 1
I20240507 08:06:09.009438 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 6 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:09.010486 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: stopping tablet replica
W20240507 08:06:09.020345 29601 connection.cc:441] server connection from 127.0.0.1:44020 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44020 (request call id 53) could send its response
I20240507 08:06:09.034916 28050 master.cc:552] Master@127.27.100.188:35355 shutdown complete.
I20240507 08:06:09.155094 30105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f03eb788ff4a4820b31171b231de9d41)
I20240507 08:06:09.155548 30105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:09.157141 30105 leader_election.cc:288] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 66b1fdcb58294d68ad651ac2fa49acb5 (127.27.100.189:36005), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:09.157845 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" candidate_term: 7 candidate_status { last_received { term: 6 index: 16 } } ignore_live_leader: false dest_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" is_pre_election: true
W20240507 08:06:09.159760 29474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.100.188:35355: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20240507 08:06:09.163326 29474 leader_election.cc:334] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355): Network error: Client connection negotiation failed: client connection to 127.27.100.188:35355: connect: Connection refused (error 111)
I20240507 08:06:09.163815 29474 leader_election.cc:302] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b50ddd4dc5d4bcb84e234822fcc7f33; no voters: 66b1fdcb58294d68ad651ac2fa49acb5, f03eb788ff4a4820b31171b231de9d41
I20240507 08:06:09.164631 30105 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:09.206230 30108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f03eb788ff4a4820b31171b231de9d41)
I20240507 08:06:09.206705 30108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:09.209527 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" candidate_term: 7 candidate_status { last_received { term: 6 index: 16 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" is_pre_election: true
I20240507 08:06:09.210243 29496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b1fdcb58294d68ad651ac2fa49acb5 in term 6.
I20240507 08:06:09.210881 30108 leader_election.cc:288] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:09.211906 29538 leader_election.cc:302] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b1fdcb58294d68ad651ac2fa49acb5, 8b50ddd4dc5d4bcb84e234822fcc7f33; no voters: 
I20240507 08:06:09.212916 30108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:09.213407 30108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader f03eb788ff4a4820b31171b231de9d41)
I20240507 08:06:09.213836 30108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:09.224284 30108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
W20240507 08:06:09.225093 29539 leader_election.cc:334] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355): Network error: Client connection negotiation failed: client connection to 127.27.100.188:35355: connect: Connection refused (error 111)
I20240507 08:06:09.229156 30108 leader_election.cc:288] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 7 election: Requested vote from peers 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177), f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355)
I20240507 08:06:09.232154 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" candidate_term: 7 candidate_status { last_received { term: 6 index: 16 } } ignore_live_leader: false dest_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33"
I20240507 08:06:09.232774 29496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:09.237969 29496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b1fdcb58294d68ad651ac2fa49acb5 in term 7.
I20240507 08:06:09.238876 29538 leader_election.cc:302] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b1fdcb58294d68ad651ac2fa49acb5, 8b50ddd4dc5d4bcb84e234822fcc7f33; no voters: 
W20240507 08:06:09.239480 29539 leader_election.cc:334] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355): Network error: Client connection negotiation failed: client connection to 127.27.100.188:35355: connect: Connection refused (error 111)
I20240507 08:06:09.239914 30108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:09.246362 30108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 7 LEADER]: Becoming Leader. State: Replica: 66b1fdcb58294d68ad651ac2fa49acb5, State: Running, Role: LEADER
I20240507 08:06:09.247130 30108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 6.16, Last appended by leader: 16, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:09.250733 30112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b1fdcb58294d68ad651ac2fa49acb5. Latest consensus state: current_term: 7 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:09.251617 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 08:06:09.251704 30112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:09.253410 30113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20240507 08:06:09.261633 30114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:09.262372 30113 catalog_manager.cc:669] Loaded metadata for table NewTableName [id=1d79b5e45252475d87377636deec0dd9]
W20240507 08:06:09.264430 30115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:09.267709 30113 tablet_loader.cc:96] loaded metadata for tablet 55e5e59c50f54e548e29ec71cabf4e69 (table NewTableName [id=1d79b5e45252475d87377636deec0dd9])
I20240507 08:06:09.268867 30113 tablet_loader.cc:96] loaded metadata for tablet 6eeffdf4714a4ad28391f055890b46ed (table NewTableName [id=1d79b5e45252475d87377636deec0dd9])
I20240507 08:06:09.270054 30113 tablet_loader.cc:96] loaded metadata for tablet 71c9b00c40c6458f9a0fe4a3a57fc99f (table NewTableName [id=1d79b5e45252475d87377636deec0dd9])
I20240507 08:06:09.271136 30113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20240507 08:06:09.276096 30117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:09.277112 30113 catalog_manager.cc:1259] Loaded cluster ID: a5e9cfaea5f8455aa6e905b407ec60c8
I20240507 08:06:09.277443 30113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:06:09.282091 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:09.283087 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:09.283352 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:09.283511 30113 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:06:09.283594 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069169283572 us; error 0 us; skew 500 ppm
I20240507 08:06:09.284484 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:09.287909 28050 webserver.cc:460] Webserver started at http://127.27.100.188:44477/ using document root <none> and password file <none>
I20240507 08:06:09.288398 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:09.288616 28050 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240507 08:06:09.288873 30113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5: Loaded TSK: 0
I20240507 08:06:09.290061 30113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:06:09.296886 28050 fs_manager.cc:709] Time spent opening directory manager: real 0.007s	user 0.007s	sys 0.000s
I20240507 08:06:09.301030 30122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:09.302033 28050 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240507 08:06:09.302435 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-2-root
uuid: "f03eb788ff4a4820b31171b231de9d41"
format_stamp: "Formatted at 2024-05-07 08:06:03 on dist-test-slave-j3cx"
I20240507 08:06:09.302992 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:09.321022 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:09.322715 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240507 08:06:09.338299 29539 consensus_peers.cc:485] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 -> Peer f03eb788ff4a4820b31171b231de9d41 (127.27.100.188:35355): Couldn't send request to peer f03eb788ff4a4820b31171b231de9d41. Status: Network error: Client connection negotiation failed: client connection to 127.27.100.188:35355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:09.342857 29496 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 7 FOLLOWER]: Refusing update from remote peer 66b1fdcb58294d68ad651ac2fa49acb5: Log matching property violated. Preceding OpId in replica: term: 6 index: 16. Preceding OpId from leader: term: 7 index: 17. (index mismatch)
I20240507 08:06:09.344594 30112 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240507 08:06:09.348131 30105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b1fdcb58294d68ad651ac2fa49acb5. Latest consensus state: current_term: 7 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:09.348906 30105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:09.355063 30108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:09.355708 30108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:09.358788 30105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:09.359580 30105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:09.397828 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.188:35355
I20240507 08:06:09.397907 30175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.188:35355 every 8 connection(s)
I20240507 08:06:09.410866 30176 sys_catalog.cc:263] Verifying existing consensus state
I20240507 08:06:09.415666 30176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: Bootstrap starting.
I20240507 08:06:09.433477 29781 heartbeater.cc:503] Master 127.27.100.189:36005 was elected leader, sending a full tablet report...
I20240507 08:06:09.435807 29859 heartbeater.cc:503] Master 127.27.100.189:36005 was elected leader, sending a full tablet report...
I20240507 08:06:09.443882 29937 heartbeater.cc:503] Master 127.27.100.189:36005 was elected leader, sending a full tablet report...
I20240507 08:06:09.470468 29936 heartbeater.cc:347] Connected to a master server at 127.27.100.188:35355
I20240507 08:06:09.495292 29552 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5: NewTableName [id=1d79b5e45252475d87377636deec0dd9] alter complete (version 1)
I20240507 08:06:09.497517 29780 heartbeater.cc:347] Connected to a master server at 127.27.100.188:35355
I20240507 08:06:09.503253 30176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=7 ignored=0} mutations{seen=12 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:06:09.504439 30176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: Bootstrap complete.
I20240507 08:06:09.507665 30176 raft_consensus.cc:348] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:09.508438 30176 raft_consensus.cc:729] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03eb788ff4a4820b31171b231de9d41, State: Initialized, Role: FOLLOWER
I20240507 08:06:09.509239 30176 consensus_queue.cc:260] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 6.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } }
I20240507 08:06:09.511258 29858 heartbeater.cc:347] Connected to a master server at 127.27.100.188:35355
I20240507 08:06:09.517359 30176 sys_catalog.cc:564] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:06:09.521924 30151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:09.570147 30112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:09.571136 30112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:09.570932 30183 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:09.572479 30183 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:09.573315 30193 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:09.574174 30193 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:09.580739 30187 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b1fdcb58294d68ad651ac2fa49acb5. Latest consensus state: current_term: 7 leader_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b50ddd4dc5d4bcb84e234822fcc7f33" member_type: VOTER last_known_addr { host: "127.27.100.190" port: 33177 } } peers { permanent_uuid: "66b1fdcb58294d68ad651ac2fa49acb5" member_type: VOTER last_known_addr { host: "127.27.100.189" port: 36005 } } peers { permanent_uuid: "f03eb788ff4a4820b31171b231de9d41" member_type: VOTER last_known_addr { host: "127.27.100.188" port: 35355 } } }
I20240507 08:06:09.581895 30187 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:06:09.597683 30199 catalog_manager.cc:1259] Loaded cluster ID: a5e9cfaea5f8455aa6e905b407ec60c8
I20240507 08:06:09.597959 30199 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: loading cluster ID for follower catalog manager: success
I20240507 08:06:09.603812 30199 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: acquiring CA information for follower catalog manager: success
I20240507 08:06:09.607396 30199 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240507 08:06:09.618368 30141 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" instance_seqno: 1715069163928510) as {username='slave'} at 127.0.0.1:44286; Asking this server to re-register.
I20240507 08:06:09.619822 29936 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:09.622864 30141 ts_manager.cc:194] Registered new tserver with Master: 64580d7ab4d84b86b6bb4a3e38fbd43f (127.27.100.131:44631)
I20240507 08:06:09.641680 30141 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" instance_seqno: 1715069163584236) as {username='slave'} at 127.0.0.1:44244; Asking this server to re-register.
I20240507 08:06:09.643162 29780 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:09.649219 30140 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3f397b9469554143884dc544bf64282a" instance_seqno: 1715069163775039) as {username='slave'} at 127.0.0.1:44270; Asking this server to re-register.
I20240507 08:06:09.651105 29858 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:09.653050 30141 ts_manager.cc:194] Registered new tserver with Master: 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455)
I20240507 08:06:09.654505 30140 ts_manager.cc:194] Registered new tserver with Master: 3f397b9469554143884dc544bf64282a (127.27.100.130:36791)
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:783: Failure
Expected equality of these values:
  live_row_count
    Which is: 67
  table_info->GetMetrics()->live_row_count->value()
    Which is: 134
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:395: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:767: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
  Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:785: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 782, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
  Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:895: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240507 08:06:39.599926 28050 tablet_server.cc:178] TabletServer@127.27.100.129:0 shutting down...
W20240507 08:06:39.626142 29798 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
W20240507 08:06:39.635788 29798 consensus_peers.cc:485] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a -> Peer 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455): Couldn't send request to peer 3cebffdfceb24f75ae9450fce8382fa5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:39.649138 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:39.649910 28050 tablet_replica.cc:330] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5: stopping tablet replica
I20240507 08:06:39.650496 28050 raft_consensus.cc:2229] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 LEADER]: Raft consensus shutting down.
I20240507 08:06:39.651553 28050 raft_consensus.cc:2258] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.653726 28050 tablet_replica.cc:330] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5: stopping tablet replica
I20240507 08:06:39.654170 28050 raft_consensus.cc:2229] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:39.654618 28050 raft_consensus.cc:2258] T 55e5e59c50f54e548e29ec71cabf4e69 P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.656384 28050 tablet_replica.cc:330] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5: stopping tablet replica
I20240507 08:06:39.656829 28050 raft_consensus.cc:2229] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:39.657285 28050 raft_consensus.cc:2258] T 6eeffdf4714a4ad28391f055890b46ed P 3cebffdfceb24f75ae9450fce8382fa5 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240507 08:06:39.700012 29798 consensus_peers.cc:485] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a -> Peer 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455): Couldn't send request to peer 3cebffdfceb24f75ae9450fce8382fa5. Status: Network error: Client connection negotiation failed: client connection to 127.27.100.129:35455: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:39.701740 28050 tablet_server.cc:194] TabletServer@127.27.100.129:0 shutdown complete.
I20240507 08:06:39.715780 28050 tablet_server.cc:178] TabletServer@127.27.100.130:0 shutting down...
I20240507 08:06:39.765619 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:39.766323 28050 tablet_replica.cc:330] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a: stopping tablet replica
I20240507 08:06:39.766937 28050 raft_consensus.cc:2229] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:39.767426 28050 raft_consensus.cc:2258] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.769383 28050 tablet_replica.cc:330] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a: stopping tablet replica
I20240507 08:06:39.769915 28050 raft_consensus.cc:2229] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 3 LEADER]: Raft consensus shutting down.
I20240507 08:06:39.770632 28050 raft_consensus.cc:2258] T 55e5e59c50f54e548e29ec71cabf4e69 P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.772517 28050 tablet_replica.cc:330] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a: stopping tablet replica
I20240507 08:06:39.772948 28050 raft_consensus.cc:2229] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 3 LEADER]: Raft consensus shutting down.
I20240507 08:06:39.773617 28050 raft_consensus.cc:2258] T 6eeffdf4714a4ad28391f055890b46ed P 3f397b9469554143884dc544bf64282a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.784636 30247 raft_consensus.cc:482] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3cebffdfceb24f75ae9450fce8382fa5)
I20240507 08:06:39.785065 30247 raft_consensus.cc:504] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cebffdfceb24f75ae9450fce8382fa5" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 35455 } } peers { permanent_uuid: "3f397b9469554143884dc544bf64282a" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 36791 } } peers { permanent_uuid: "64580d7ab4d84b86b6bb4a3e38fbd43f" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 44631 } }
I20240507 08:06:39.786935 30247 leader_election.cc:288] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455), 3f397b9469554143884dc544bf64282a (127.27.100.130:36791)
W20240507 08:06:39.794309 29876 leader_election.cc:334] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3cebffdfceb24f75ae9450fce8382fa5 (127.27.100.129:35455): Network error: Client connection negotiation failed: client connection to 127.27.100.129:35455: connect: Connection refused (error 111)
I20240507 08:06:39.797348 28050 tablet_server.cc:194] TabletServer@127.27.100.130:0 shutdown complete.
W20240507 08:06:39.798343 29877 leader_election.cc:334] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3f397b9469554143884dc544bf64282a (127.27.100.130:36791): Network error: Client connection negotiation failed: client connection to 127.27.100.130:36791: connect: Connection refused (error 111)
I20240507 08:06:39.798760 29877 leader_election.cc:302] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64580d7ab4d84b86b6bb4a3e38fbd43f; no voters: 3cebffdfceb24f75ae9450fce8382fa5, 3f397b9469554143884dc544bf64282a
I20240507 08:06:39.799486 30247 raft_consensus.cc:2726] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:39.811606 28050 tablet_server.cc:178] TabletServer@127.27.100.131:0 shutting down...
I20240507 08:06:39.841075 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:39.841650 28050 tablet_replica.cc:330] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f: stopping tablet replica
I20240507 08:06:39.842520 28050 raft_consensus.cc:2229] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:39.842988 28050 raft_consensus.cc:2258] T 55e5e59c50f54e548e29ec71cabf4e69 P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.844956 28050 tablet_replica.cc:330] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f: stopping tablet replica
I20240507 08:06:39.845412 28050 raft_consensus.cc:2229] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:39.845881 28050 raft_consensus.cc:2258] T 6eeffdf4714a4ad28391f055890b46ed P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.847631 28050 tablet_replica.cc:330] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f: stopping tablet replica
I20240507 08:06:39.848062 28050 raft_consensus.cc:2229] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:39.848488 28050 raft_consensus.cc:2258] T 71c9b00c40c6458f9a0fe4a3a57fc99f P 64580d7ab4d84b86b6bb4a3e38fbd43f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.870357 28050 tablet_server.cc:194] TabletServer@127.27.100.131:0 shutdown complete.
I20240507 08:06:39.885331 28050 master.cc:537] Master@127.27.100.190:33177 shutting down...
I20240507 08:06:39.910288 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:39.911062 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.911386 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8b50ddd4dc5d4bcb84e234822fcc7f33: stopping tablet replica
I20240507 08:06:39.929682 28050 master.cc:552] Master@127.27.100.190:33177 shutdown complete.
I20240507 08:06:39.947729 28050 master.cc:537] Master@127.27.100.189:36005 shutting down...
W20240507 08:06:39.956987 29538 consensus_peers.cc:485] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 -> Peer 8b50ddd4dc5d4bcb84e234822fcc7f33 (127.27.100.190:33177): Couldn't send request to peer 8b50ddd4dc5d4bcb84e234822fcc7f33. Status: Network error: Client connection negotiation failed: client connection to 127.27.100.190:33177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:39.969559 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 7 LEADER]: Raft consensus shutting down.
I20240507 08:06:39.970486 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:39.970820 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P 66b1fdcb58294d68ad651ac2fa49acb5: stopping tablet replica
I20240507 08:06:39.990850 28050 master.cc:552] Master@127.27.100.189:36005 shutdown complete.
I20240507 08:06:40.010131 28050 master.cc:537] Master@127.27.100.188:35355 shutting down...
I20240507 08:06:40.026506 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:40.027032 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f03eb788ff4a4820b31171b231de9d41 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:40.027289 28050 tablet_replica.cc:330] stopping tablet replica
I20240507 08:06:40.044183 28050 master.cc:552] Master@127.27.100.188:35355 shutdown complete.
I20240507 08:06:40.069330 28050 test_util.cc:175] -----------------------------------------------
I20240507 08:06:40.069541 28050 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1715069146555183-28050-0
[  FAILED  ] TsTabletManagerITest.TestTableStats (37025 ms)
[ RUN      ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20240507 08:06:40.075665 28050 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.100.190:45577
I20240507 08:06:40.076727 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:40.083695 30257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:40.084913 30258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:40.085806 28050 server_base.cc:1034] running on GCE node
W20240507 08:06:40.086797 30260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:40.088882 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:40.089082 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:40.089241 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069200089221 us; error 0 us; skew 500 ppm
I20240507 08:06:40.089754 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:40.092232 28050 webserver.cc:460] Webserver started at http://127.27.100.190:35105/ using document root <none> and password file <none>
I20240507 08:06:40.092706 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:40.092895 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:40.093158 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:40.094414 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/master-0-root/instance:
uuid: "0968128e54f54bf5953b6c336585737e"
format_stamp: "Formatted at 2024-05-07 08:06:40 on dist-test-slave-j3cx"
I20240507 08:06:40.098757 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240507 08:06:40.101994 30265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:40.102779 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240507 08:06:40.103039 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/master-0-root
uuid: "0968128e54f54bf5953b6c336585737e"
format_stamp: "Formatted at 2024-05-07 08:06:40 on dist-test-slave-j3cx"
I20240507 08:06:40.103307 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:40.122774 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:40.124058 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:40.160573 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.190:45577
I20240507 08:06:40.160661 30316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.190:45577 every 8 connection(s)
I20240507 08:06:40.164454 30317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:40.177412 30317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e: Bootstrap starting.
I20240507 08:06:40.181623 30317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:40.185750 30317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e: No bootstrap required, opened a new log
I20240507 08:06:40.188051 30317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0968128e54f54bf5953b6c336585737e" member_type: VOTER }
I20240507 08:06:40.188645 30317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:40.188959 30317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0968128e54f54bf5953b6c336585737e, State: Initialized, Role: FOLLOWER
I20240507 08:06:40.189695 30317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0968128e54f54bf5953b6c336585737e" member_type: VOTER }
I20240507 08:06:40.190412 30317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:06:40.190737 30317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:06:40.191087 30317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:40.196885 30317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0968128e54f54bf5953b6c336585737e" member_type: VOTER }
I20240507 08:06:40.197369 30317 leader_election.cc:302] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0968128e54f54bf5953b6c336585737e; no voters: 
I20240507 08:06:40.198437 30317 leader_election.cc:288] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:06:40.198760 30320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:40.200110 30320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [term 1 LEADER]: Becoming Leader. State: Replica: 0968128e54f54bf5953b6c336585737e, State: Running, Role: LEADER
I20240507 08:06:40.200863 30320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0968128e54f54bf5953b6c336585737e" member_type: VOTER }
I20240507 08:06:40.201349 30317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:06:40.203896 30322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0968128e54f54bf5953b6c336585737e. Latest consensus state: current_term: 1 leader_uuid: "0968128e54f54bf5953b6c336585737e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0968128e54f54bf5953b6c336585737e" member_type: VOTER } }
I20240507 08:06:40.204468 30322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:40.204105 30321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0968128e54f54bf5953b6c336585737e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0968128e54f54bf5953b6c336585737e" member_type: VOTER } }
I20240507 08:06:40.205114 30321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:40.213413 30325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:06:40.218091 30325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:06:40.219200 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:06:40.225878 30325 catalog_manager.cc:1347] Generated new cluster ID: 17cf689f83a24d80b406492df73b00e7
I20240507 08:06:40.226168 30325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:06:40.244680 30325 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:06:40.245923 30325 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:06:40.256361 30325 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e: Generated new TSK 0
I20240507 08:06:40.256917 30325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:06:40.285595 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:40.292486 30338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:40.293833 30339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:40.294524 30341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:40.295360 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:40.296267 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:40.296465 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:40.296612 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069200296595 us; error 0 us; skew 500 ppm
I20240507 08:06:40.297125 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:40.299434 28050 webserver.cc:460] Webserver started at http://127.27.100.129:35647/ using document root <none> and password file <none>
I20240507 08:06:40.299952 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:40.300132 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:40.300379 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:40.301437 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-0-root/instance:
uuid: "98b072741c8d4d139b07484901251913"
format_stamp: "Formatted at 2024-05-07 08:06:40 on dist-test-slave-j3cx"
I20240507 08:06:40.305902 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240507 08:06:40.309132 30346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:40.309870 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240507 08:06:40.310133 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "98b072741c8d4d139b07484901251913"
format_stamp: "Formatted at 2024-05-07 08:06:40 on dist-test-slave-j3cx"
I20240507 08:06:40.310426 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:40.329180 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:40.330303 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:40.331794 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:40.334039 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:40.334242 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:40.334466 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:40.334630 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:40.373411 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:42551
I20240507 08:06:40.373497 30408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:42551 every 8 connection(s)
I20240507 08:06:40.378551 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:40.387004 30413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:40.388242 30414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:40.390535 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:40.391028 30409 heartbeater.cc:347] Connected to a master server at 127.27.100.190:45577
I20240507 08:06:40.391391 30409 heartbeater.cc:464] Registering TS with master...
W20240507 08:06:40.391458 30416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:40.392189 30409 heartbeater.cc:511] Master 127.27.100.190:45577 requested a full tablet report, sending...
I20240507 08:06:40.392402 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:40.392637 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:40.392788 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069200392773 us; error 0 us; skew 500 ppm
I20240507 08:06:40.393378 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:40.394069 30282 ts_manager.cc:194] Registered new tserver with Master: 98b072741c8d4d139b07484901251913 (127.27.100.129:42551)
I20240507 08:06:40.395879 30282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57172
I20240507 08:06:40.398243 28050 webserver.cc:460] Webserver started at http://127.27.100.130:41781/ using document root <none> and password file <none>
I20240507 08:06:40.398730 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:40.398928 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:40.399178 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:40.400265 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-1-root/instance:
uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43"
format_stamp: "Formatted at 2024-05-07 08:06:40 on dist-test-slave-j3cx"
I20240507 08:06:40.404515 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.004s
I20240507 08:06:40.407747 30421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:40.408483 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.003s
I20240507 08:06:40.408764 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-1-root
uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43"
format_stamp: "Formatted at 2024-05-07 08:06:40 on dist-test-slave-j3cx"
I20240507 08:06:40.409034 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:40.419394 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:40.420506 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:40.421774 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:40.423887 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:40.424086 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:40.424294 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:40.424443 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:40.465065 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.130:45971
I20240507 08:06:40.465160 30483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.130:45971 every 8 connection(s)
I20240507 08:06:40.469694 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:40.477766 30487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:40.478652 30488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:40.481200 30490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:40.481729 30484 heartbeater.cc:347] Connected to a master server at 127.27.100.190:45577
I20240507 08:06:40.482115 30484 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:40.482473 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:40.483078 30484 heartbeater.cc:511] Master 127.27.100.190:45577 requested a full tablet report, sending...
I20240507 08:06:40.483412 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:40.483716 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:40.483922 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069200483905 us; error 0 us; skew 500 ppm
I20240507 08:06:40.484723 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:40.485214 30282 ts_manager.cc:194] Registered new tserver with Master: f4f8a57d7ac24d5780e0d2953ff2bd43 (127.27.100.130:45971)
I20240507 08:06:40.486729 30282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57184
I20240507 08:06:40.487334 28050 webserver.cc:460] Webserver started at http://127.27.100.131:41729/ using document root <none> and password file <none>
I20240507 08:06:40.488024 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:40.488299 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:40.488629 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:40.490093 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-2-root/instance:
uuid: "31d8e710df874fb8bd039866696822dd"
format_stamp: "Formatted at 2024-05-07 08:06:40 on dist-test-slave-j3cx"
I20240507 08:06:40.494788 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.007s	sys 0.000s
I20240507 08:06:40.497915 30495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:40.498679 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240507 08:06:40.498972 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-2-root
uuid: "31d8e710df874fb8bd039866696822dd"
format_stamp: "Formatted at 2024-05-07 08:06:40 on dist-test-slave-j3cx"
I20240507 08:06:40.499256 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:40.521618 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:40.522754 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:40.524205 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:40.526321 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:40.526516 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:40.526744 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:40.526897 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:40.563872 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.131:39537
I20240507 08:06:40.563961 30557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.131:39537 every 8 connection(s)
I20240507 08:06:40.576462 30558 heartbeater.cc:347] Connected to a master server at 127.27.100.190:45577
I20240507 08:06:40.576819 30558 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:40.577464 30558 heartbeater.cc:511] Master 127.27.100.190:45577 requested a full tablet report, sending...
I20240507 08:06:40.579164 30282 ts_manager.cc:194] Registered new tserver with Master: 31d8e710df874fb8bd039866696822dd (127.27.100.131:39537)
I20240507 08:06:40.579262 28050 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012416589s
I20240507 08:06:40.580446 30282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57190
I20240507 08:06:40.612656 30282 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57192:
name: "test-table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
  }
}
W20240507 08:06:40.614153 30282 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:06:40.656297 30523 tablet_service.cc:1467] Processing CreateTablet for tablet 46e7cd19bc0b4470bd6c34829cfd4494 (DEFAULT_TABLE table=test-table [id=21bae3297cfb460883462683829394db]), partition=
I20240507 08:06:40.657232 30523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e7cd19bc0b4470bd6c34829cfd4494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:40.661324 30449 tablet_service.cc:1467] Processing CreateTablet for tablet 46e7cd19bc0b4470bd6c34829cfd4494 (DEFAULT_TABLE table=test-table [id=21bae3297cfb460883462683829394db]), partition=
I20240507 08:06:40.662431 30449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e7cd19bc0b4470bd6c34829cfd4494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:40.664544 30374 tablet_service.cc:1467] Processing CreateTablet for tablet 46e7cd19bc0b4470bd6c34829cfd4494 (DEFAULT_TABLE table=test-table [id=21bae3297cfb460883462683829394db]), partition=
I20240507 08:06:40.665614 30374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e7cd19bc0b4470bd6c34829cfd4494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:40.674217 30574 tablet_bootstrap.cc:492] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd: Bootstrap starting.
I20240507 08:06:40.678063 30575 tablet_bootstrap.cc:492] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Bootstrap starting.
I20240507 08:06:40.679610 30574 tablet_bootstrap.cc:654] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:40.684561 30575 tablet_bootstrap.cc:654] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:40.688696 30576 tablet_bootstrap.cc:492] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913: Bootstrap starting.
I20240507 08:06:40.691008 30574 tablet_bootstrap.cc:492] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd: No bootstrap required, opened a new log
I20240507 08:06:40.691156 30575 tablet_bootstrap.cc:492] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: No bootstrap required, opened a new log
I20240507 08:06:40.691545 30574 ts_tablet_manager.cc:1367] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.011s
I20240507 08:06:40.691644 30575 ts_tablet_manager.cc:1367] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Time spent bootstrapping tablet: real 0.014s	user 0.014s	sys 0.000s
I20240507 08:06:40.693917 30575 raft_consensus.cc:348] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b072741c8d4d139b07484901251913" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 42551 } } peers { permanent_uuid: "31d8e710df874fb8bd039866696822dd" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39537 } } peers { permanent_uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 45971 } }
I20240507 08:06:40.694689 30575 raft_consensus.cc:374] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:40.694404 30574 raft_consensus.cc:348] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b072741c8d4d139b07484901251913" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 42551 } } peers { permanent_uuid: "31d8e710df874fb8bd039866696822dd" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39537 } } peers { permanent_uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 45971 } }
I20240507 08:06:40.695015 30575 raft_consensus.cc:729] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f8a57d7ac24d5780e0d2953ff2bd43, State: Initialized, Role: FOLLOWER
I20240507 08:06:40.695103 30576 tablet_bootstrap.cc:654] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:40.695226 30574 raft_consensus.cc:374] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:40.695632 30574 raft_consensus.cc:729] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d8e710df874fb8bd039866696822dd, State: Initialized, Role: FOLLOWER
I20240507 08:06:40.695879 30575 consensus_queue.cc:260] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b072741c8d4d139b07484901251913" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 42551 } } peers { permanent_uuid: "31d8e710df874fb8bd039866696822dd" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39537 } } peers { permanent_uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 45971 } }
I20240507 08:06:40.696475 30574 consensus_queue.cc:260] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b072741c8d4d139b07484901251913" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 42551 } } peers { permanent_uuid: "31d8e710df874fb8bd039866696822dd" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39537 } } peers { permanent_uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 45971 } }
I20240507 08:06:40.698401 30484 heartbeater.cc:503] Master 127.27.100.190:45577 was elected leader, sending a full tablet report...
I20240507 08:06:40.698876 30575 ts_tablet_manager.cc:1398] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
W20240507 08:06:40.699184 30575 ts_tablet_manager.cc:1420] Injecting 50ms of latency into OpenTablet
I20240507 08:06:40.701107 30558 heartbeater.cc:503] Master 127.27.100.190:45577 was elected leader, sending a full tablet report...
I20240507 08:06:40.701400 30574 ts_tablet_manager.cc:1398] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.004s
W20240507 08:06:40.701653 30574 ts_tablet_manager.cc:1420] Injecting 50ms of latency into OpenTablet
I20240507 08:06:40.702610 30576 tablet_bootstrap.cc:492] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913: No bootstrap required, opened a new log
I20240507 08:06:40.703078 30576 ts_tablet_manager.cc:1367] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913: Time spent bootstrapping tablet: real 0.015s	user 0.008s	sys 0.004s
I20240507 08:06:40.705510 30576 raft_consensus.cc:348] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b072741c8d4d139b07484901251913" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 42551 } } peers { permanent_uuid: "31d8e710df874fb8bd039866696822dd" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39537 } } peers { permanent_uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 45971 } }
I20240507 08:06:40.706182 30576 raft_consensus.cc:374] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:40.706408 30576 raft_consensus.cc:729] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98b072741c8d4d139b07484901251913, State: Initialized, Role: FOLLOWER
I20240507 08:06:40.706902 30576 consensus_queue.cc:260] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b072741c8d4d139b07484901251913" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 42551 } } peers { permanent_uuid: "31d8e710df874fb8bd039866696822dd" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39537 } } peers { permanent_uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 45971 } }
I20240507 08:06:40.708496 30409 heartbeater.cc:503] Master 127.27.100.190:45577 was elected leader, sending a full tablet report...
I20240507 08:06:40.708937 30576 ts_tablet_manager.cc:1398] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.004s
W20240507 08:06:40.709218 30576 ts_tablet_manager.cc:1420] Injecting 50ms of latency into OpenTablet
W20240507 08:06:40.718842 30485 tablet.cc:2369] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 08:06:40.830024 30582 raft_consensus.cc:482] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:40.830562 30582 raft_consensus.cc:504] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b072741c8d4d139b07484901251913" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 42551 } } peers { permanent_uuid: "31d8e710df874fb8bd039866696822dd" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39537 } } peers { permanent_uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 45971 } }
I20240507 08:06:40.832446 30582 leader_election.cc:288] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31d8e710df874fb8bd039866696822dd (127.27.100.131:39537), f4f8a57d7ac24d5780e0d2953ff2bd43 (127.27.100.130:45971)
I20240507 08:06:40.840204 30533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e7cd19bc0b4470bd6c34829cfd4494" candidate_uuid: "98b072741c8d4d139b07484901251913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d8e710df874fb8bd039866696822dd" is_pre_election: true
I20240507 08:06:40.840993 30533 raft_consensus.cc:2454] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98b072741c8d4d139b07484901251913 in term 0.
I20240507 08:06:40.842015 30459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e7cd19bc0b4470bd6c34829cfd4494" candidate_uuid: "98b072741c8d4d139b07484901251913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43" is_pre_election: true
I20240507 08:06:40.842200 30347 leader_election.cc:302] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d8e710df874fb8bd039866696822dd, 98b072741c8d4d139b07484901251913; no voters: 
I20240507 08:06:40.842667 30459 raft_consensus.cc:2454] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98b072741c8d4d139b07484901251913 in term 0.
I20240507 08:06:40.843065 30582 raft_consensus.cc:2781] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:06:40.843389 30582 raft_consensus.cc:482] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:40.843744 30582 raft_consensus.cc:3037] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:40.848071 30582 raft_consensus.cc:504] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b072741c8d4d139b07484901251913" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 42551 } } peers { permanent_uuid: "31d8e710df874fb8bd039866696822dd" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39537 } } peers { permanent_uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 45971 } }
I20240507 08:06:40.849475 30582 leader_election.cc:288] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [CANDIDATE]: Term 1 election: Requested vote from peers 31d8e710df874fb8bd039866696822dd (127.27.100.131:39537), f4f8a57d7ac24d5780e0d2953ff2bd43 (127.27.100.130:45971)
I20240507 08:06:40.850347 30533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e7cd19bc0b4470bd6c34829cfd4494" candidate_uuid: "98b072741c8d4d139b07484901251913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d8e710df874fb8bd039866696822dd"
I20240507 08:06:40.850467 30459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e7cd19bc0b4470bd6c34829cfd4494" candidate_uuid: "98b072741c8d4d139b07484901251913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43"
I20240507 08:06:40.850839 30533 raft_consensus.cc:3037] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:40.850963 30459 raft_consensus.cc:3037] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:40.855340 30459 raft_consensus.cc:2454] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98b072741c8d4d139b07484901251913 in term 1.
I20240507 08:06:40.855350 30533 raft_consensus.cc:2454] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98b072741c8d4d139b07484901251913 in term 1.
I20240507 08:06:40.856487 30347 leader_election.cc:302] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98b072741c8d4d139b07484901251913, f4f8a57d7ac24d5780e0d2953ff2bd43; no voters: 
I20240507 08:06:40.857092 30582 raft_consensus.cc:2781] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:40.857872 30582 raft_consensus.cc:686] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [term 1 LEADER]: Becoming Leader. State: Replica: 98b072741c8d4d139b07484901251913, State: Running, Role: LEADER
I20240507 08:06:40.858610 30582 consensus_queue.cc:237] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b072741c8d4d139b07484901251913" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 42551 } } peers { permanent_uuid: "31d8e710df874fb8bd039866696822dd" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39537 } } peers { permanent_uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 45971 } }
I20240507 08:06:40.864338 30282 catalog_manager.cc:5484] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98b072741c8d4d139b07484901251913 (127.27.100.129). New cstate: current_term: 1 leader_uuid: "98b072741c8d4d139b07484901251913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b072741c8d4d139b07484901251913" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 42551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d8e710df874fb8bd039866696822dd" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 45971 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:40.913506 30449 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e7cd19bc0b4470bd6c34829cfd4494 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:33234
I20240507 08:06:40.914979 30587 tablet_replica.cc:330] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: stopping tablet replica
I20240507 08:06:40.915593 30587 raft_consensus.cc:2229] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:40.916005 30587 raft_consensus.cc:2258] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:40.917932 30587 ts_tablet_manager.cc:1875] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:06:40.927654 30587 ts_tablet_manager.cc:1888] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 0.0
I20240507 08:06:40.927970 30587 log.cc:1196] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Deleting WAL directory at /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-1-root/wals/46e7cd19bc0b4470bd6c34829cfd4494
I20240507 08:06:40.931588 30588 ts_tablet_manager.cc:902] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Initiating tablet copy from peer 98b072741c8d4d139b07484901251913 (127.27.100.129:42551)
I20240507 08:06:40.933211 30588 tablet_copy_client.cc:316] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: tablet copy: Beginning tablet copy session from remote peer at address 127.27.100.129:42551
I20240507 08:06:40.940727 30394 tablet_copy_service.cc:140] P 98b072741c8d4d139b07484901251913: Received BeginTabletCopySession request for tablet 46e7cd19bc0b4470bd6c34829cfd4494 from peer f4f8a57d7ac24d5780e0d2953ff2bd43 ({username='slave'} at 127.0.0.1:44202)
I20240507 08:06:40.941203 30394 tablet_copy_service.cc:161] P 98b072741c8d4d139b07484901251913: Beginning new tablet copy session on tablet 46e7cd19bc0b4470bd6c34829cfd4494 from peer f4f8a57d7ac24d5780e0d2953ff2bd43 at {username='slave'} at 127.0.0.1:44202: session id = f4f8a57d7ac24d5780e0d2953ff2bd43-46e7cd19bc0b4470bd6c34829cfd4494
I20240507 08:06:40.946480 30394 tablet_copy_source_session.cc:205] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:06:40.948989 30588 ts_tablet_manager.cc:1875] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240507 08:06:40.958443 30588 ts_tablet_manager.cc:1888] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 0.0
I20240507 08:06:40.959064 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e7cd19bc0b4470bd6c34829cfd4494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:40.963889 30588 tablet_copy_client.cc:791] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: tablet copy: Starting download of 0 data blocks...
I20240507 08:06:40.964421 30588 tablet_copy_client.cc:655] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: tablet copy: Starting download of 1 WAL segments...
I20240507 08:06:40.966099 30282 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57194:
table { table_id: "21bae3297cfb460883462683829394db" table_name: "test-table" }
I20240507 08:06:40.966672 30282 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57194:
table { table_id: "21bae3297cfb460883462683829394db" table_name: "test-table" }
I20240507 08:06:40.968155 30588 tablet_copy_client.cc:531] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:06:40.975278 30588 tablet_bootstrap.cc:492] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Bootstrap starting.
I20240507 08:06:40.977180 30282 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e: Sending DeleteTablet for 3 replicas of tablet 46e7cd19bc0b4470bd6c34829cfd4494
I20240507 08:06:40.979054 30374 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e7cd19bc0b4470bd6c34829cfd4494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:06:40 UTC) from {username='slave'} at 127.0.0.1:44196
I20240507 08:06:40.979693 30523 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e7cd19bc0b4470bd6c34829cfd4494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:06:40 UTC) from {username='slave'} at 127.0.0.1:49144
I20240507 08:06:40.979831 30449 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e7cd19bc0b4470bd6c34829cfd4494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:06:40 UTC) from {username='slave'} at 127.0.0.1:33224
I20240507 08:06:40.983687 30590 tablet_replica.cc:330] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913: stopping tablet replica
W20240507 08:06:40.983848 30266 catalog_manager.cc:4810] TS f4f8a57d7ac24d5780e0d2953ff2bd43 (127.27.100.130:45971): delete failed for tablet 46e7cd19bc0b4470bd6c34829cfd4494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46e7cd19bc0b4470bd6c34829cfd4494 already in progress: copying tablet
I20240507 08:06:40.984892 30590 raft_consensus.cc:2229] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:40.985339 30591 tablet_replica.cc:330] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd: stopping tablet replica
I20240507 08:06:40.985821 30590 pending_rounds.cc:62] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913: Trying to abort 1 pending ops.
I20240507 08:06:40.986151 30590 pending_rounds.cc:68] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024923446723637248 op_type: NO_OP noop_request { }
I20240507 08:06:40.986529 30590 raft_consensus.cc:2866] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:06:40.986821 30590 raft_consensus.cc:2258] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:40.987077 30591 raft_consensus.cc:2229] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:40.988240 30591 raft_consensus.cc:2258] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:40.989867 30590 ts_tablet_manager.cc:1875] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:06:40.990193 30591 ts_tablet_manager.cc:1875] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:06:40.991773 30588 tablet_bootstrap.cc:492] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240507 08:06:40.992738 30588 tablet_bootstrap.cc:492] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Bootstrap complete.
I20240507 08:06:40.993438 30588 ts_tablet_manager.cc:1367] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Time spent bootstrapping tablet: real 0.018s	user 0.007s	sys 0.012s
I20240507 08:06:40.996279 30588 raft_consensus.cc:348] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b072741c8d4d139b07484901251913" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 42551 } } peers { permanent_uuid: "31d8e710df874fb8bd039866696822dd" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39537 } } peers { permanent_uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 45971 } }
I20240507 08:06:40.997241 30588 raft_consensus.cc:729] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f8a57d7ac24d5780e0d2953ff2bd43, State: Initialized, Role: FOLLOWER
I20240507 08:06:40.998081 30588 consensus_queue.cc:260] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98b072741c8d4d139b07484901251913" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 42551 } } peers { permanent_uuid: "31d8e710df874fb8bd039866696822dd" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 39537 } } peers { permanent_uuid: "f4f8a57d7ac24d5780e0d2953ff2bd43" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 45971 } }
I20240507 08:06:41.000272 30588 ts_tablet_manager.cc:1398] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.002s
W20240507 08:06:41.000572 30588 ts_tablet_manager.cc:1420] Injecting 50ms of latency into OpenTablet
I20240507 08:06:41.002480 30591 ts_tablet_manager.cc:1888] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:06:41.002810 30591 log.cc:1196] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd: Deleting WAL directory at /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-2-root/wals/46e7cd19bc0b4470bd6c34829cfd4494
I20240507 08:06:41.003407 30591 ts_tablet_manager.cc:1909] T 46e7cd19bc0b4470bd6c34829cfd4494 P 31d8e710df874fb8bd039866696822dd: Deleting consensus metadata
I20240507 08:06:41.004850 30590 ts_tablet_manager.cc:1888] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:06:41.005231 30590 log.cc:1196] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913: Deleting WAL directory at /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-0-root/wals/46e7cd19bc0b4470bd6c34829cfd4494
I20240507 08:06:41.005944 30590 ts_tablet_manager.cc:1909] T 46e7cd19bc0b4470bd6c34829cfd4494 P 98b072741c8d4d139b07484901251913: Deleting consensus metadata
I20240507 08:06:41.006443 30449 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e7cd19bc0b4470bd6c34829cfd4494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:06:40 UTC) from {username='slave'} at 127.0.0.1:33224
I20240507 08:06:41.007015 30266 catalog_manager.cc:4830] TS 31d8e710df874fb8bd039866696822dd (127.27.100.131:39537): tablet 46e7cd19bc0b4470bd6c34829cfd4494 (table test-table [id=21bae3297cfb460883462683829394db]) successfully deleted
I20240507 08:06:41.007772 30267 catalog_manager.cc:4830] TS 98b072741c8d4d139b07484901251913 (127.27.100.129:42551): tablet 46e7cd19bc0b4470bd6c34829cfd4494 (table test-table [id=21bae3297cfb460883462683829394db]) successfully deleted
W20240507 08:06:41.007889 30266 catalog_manager.cc:4810] TS f4f8a57d7ac24d5780e0d2953ff2bd43 (127.27.100.130:45971): delete failed for tablet 46e7cd19bc0b4470bd6c34829cfd4494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46e7cd19bc0b4470bd6c34829cfd4494 already in progress: copying tablet
I20240507 08:06:41.052722 30394 tablet_copy_service.cc:342] P 98b072741c8d4d139b07484901251913: Request end of tablet copy session f4f8a57d7ac24d5780e0d2953ff2bd43-46e7cd19bc0b4470bd6c34829cfd4494 received from {username='slave'} at 127.0.0.1:44202
I20240507 08:06:41.053181 30394 tablet_copy_service.cc:433] P 98b072741c8d4d139b07484901251913: ending tablet copy session f4f8a57d7ac24d5780e0d2953ff2bd43-46e7cd19bc0b4470bd6c34829cfd4494 on tablet 46e7cd19bc0b4470bd6c34829cfd4494 with peer f4f8a57d7ac24d5780e0d2953ff2bd43
I20240507 08:06:41.056243 30449 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e7cd19bc0b4470bd6c34829cfd4494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:06:40 UTC) from {username='slave'} at 127.0.0.1:33224
I20240507 08:06:41.057011 30587 tablet_replica.cc:330] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: stopping tablet replica
I20240507 08:06:41.057579 30587 raft_consensus.cc:2229] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:41.057962 30587 pending_rounds.cc:62] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Trying to abort 1 pending ops.
I20240507 08:06:41.058163 30587 pending_rounds.cc:68] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024923446723637248 op_type: NO_OP noop_request { }
I20240507 08:06:41.058473 30587 raft_consensus.cc:2866] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:06:41.058832 30587 raft_consensus.cc:2258] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:41.060849 30587 ts_tablet_manager.cc:1875] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:06:41.070400 30587 ts_tablet_manager.cc:1888] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:06:41.070716 30587 log.cc:1196] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Deleting WAL directory at /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1715069146555183-28050-0/minicluster-data/ts-1-root/wals/46e7cd19bc0b4470bd6c34829cfd4494
I20240507 08:06:41.071296 30587 ts_tablet_manager.cc:1909] T 46e7cd19bc0b4470bd6c34829cfd4494 P f4f8a57d7ac24d5780e0d2953ff2bd43: Deleting consensus metadata
I20240507 08:06:41.073623 30266 catalog_manager.cc:4830] TS f4f8a57d7ac24d5780e0d2953ff2bd43 (127.27.100.130:45971): tablet 46e7cd19bc0b4470bd6c34829cfd4494 (table test-table [id=21bae3297cfb460883462683829394db]) successfully deleted
I20240507 08:06:41.114244 28050 tablet_server.cc:178] TabletServer@127.27.100.129:0 shutting down...
I20240507 08:06:41.132275 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:41.149529 28050 tablet_server.cc:194] TabletServer@127.27.100.129:0 shutdown complete.
I20240507 08:06:41.158169 28050 tablet_server.cc:178] TabletServer@127.27.100.130:0 shutting down...
I20240507 08:06:41.175146 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:41.191447 28050 tablet_server.cc:194] TabletServer@127.27.100.130:0 shutdown complete.
I20240507 08:06:41.200501 28050 tablet_server.cc:178] TabletServer@127.27.100.131:0 shutting down...
I20240507 08:06:41.216645 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:41.232962 28050 tablet_server.cc:194] TabletServer@127.27.100.131:0 shutdown complete.
I20240507 08:06:41.241281 28050 master.cc:537] Master@127.27.100.190:45577 shutting down...
I20240507 08:06:41.254994 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:41.255483 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:41.255808 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0968128e54f54bf5953b6c336585737e: stopping tablet replica
I20240507 08:06:41.273990 28050 master.cc:552] Master@127.27.100.190:45577 shutdown complete.
[       OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (1222 ms)
[----------] 4 tests from TsTabletManagerITest (44293 ms total)

[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN      ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20240507 08:06:41.298048 28050 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.100.190:45831
I20240507 08:06:41.299103 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:41.305061 30597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:41.306874 30600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:41.306967 30598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:41.307936 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:41.308570 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:41.308740 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:41.308853 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069201308842 us; error 0 us; skew 500 ppm
I20240507 08:06:41.309334 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:41.311800 28050 webserver.cc:460] Webserver started at http://127.27.100.190:42733/ using document root <none> and password file <none>
I20240507 08:06:41.312227 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:41.312376 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:41.312583 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:41.313638 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/master-0-root/instance:
uuid: "b4acf736ba1d4afb8ad1dbc913392575"
format_stamp: "Formatted at 2024-05-07 08:06:41 on dist-test-slave-j3cx"
I20240507 08:06:41.318001 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240507 08:06:41.321298 30605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:41.322069 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240507 08:06:41.322351 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/master-0-root
uuid: "b4acf736ba1d4afb8ad1dbc913392575"
format_stamp: "Formatted at 2024-05-07 08:06:41 on dist-test-slave-j3cx"
I20240507 08:06:41.322638 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:41.342923 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:41.343961 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:41.378346 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.190:45831
I20240507 08:06:41.378474 30656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.190:45831 every 8 connection(s)
I20240507 08:06:41.382076 30657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:41.392252 30657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575: Bootstrap starting.
I20240507 08:06:41.396706 30657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:41.400672 30657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575: No bootstrap required, opened a new log
I20240507 08:06:41.402508 30657 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4acf736ba1d4afb8ad1dbc913392575" member_type: VOTER }
I20240507 08:06:41.402915 30657 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:41.403143 30657 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4acf736ba1d4afb8ad1dbc913392575, State: Initialized, Role: FOLLOWER
I20240507 08:06:41.403748 30657 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4acf736ba1d4afb8ad1dbc913392575" member_type: VOTER }
I20240507 08:06:41.404202 30657 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:06:41.404413 30657 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:06:41.404675 30657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:41.408963 30657 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4acf736ba1d4afb8ad1dbc913392575" member_type: VOTER }
I20240507 08:06:41.409458 30657 leader_election.cc:302] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4acf736ba1d4afb8ad1dbc913392575; no voters: 
I20240507 08:06:41.410528 30657 leader_election.cc:288] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:06:41.410849 30660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:41.412205 30660 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [term 1 LEADER]: Becoming Leader. State: Replica: b4acf736ba1d4afb8ad1dbc913392575, State: Running, Role: LEADER
I20240507 08:06:41.412860 30660 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4acf736ba1d4afb8ad1dbc913392575" member_type: VOTER }
I20240507 08:06:41.413425 30657 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:06:41.417276 30662 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4acf736ba1d4afb8ad1dbc913392575. Latest consensus state: current_term: 1 leader_uuid: "b4acf736ba1d4afb8ad1dbc913392575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4acf736ba1d4afb8ad1dbc913392575" member_type: VOTER } }
I20240507 08:06:41.417964 30662 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:41.419835 30661 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4acf736ba1d4afb8ad1dbc913392575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4acf736ba1d4afb8ad1dbc913392575" member_type: VOTER } }
I20240507 08:06:41.420550 30661 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:41.425858 30668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:06:41.431286 30668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:06:41.432389 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:06:41.439469 30668 catalog_manager.cc:1347] Generated new cluster ID: 19b1a9659cfb427c9966fe776879a179
I20240507 08:06:41.439814 30668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:06:41.456115 30668 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:06:41.457337 30668 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:06:41.471809 30668 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575: Generated new TSK 0
I20240507 08:06:41.472473 30668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:06:41.499045 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:41.505834 30678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:41.506736 30679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:41.507936 30681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:41.508857 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:41.509588 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:41.509814 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:41.509971 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069201509956 us; error 0 us; skew 500 ppm
I20240507 08:06:41.510512 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:41.512887 28050 webserver.cc:460] Webserver started at http://127.27.100.129:38113/ using document root <none> and password file <none>
I20240507 08:06:41.513357 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:41.513532 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:41.513808 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:41.514958 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root/instance:
uuid: "b06daaf3dab0462a8f21abec33a5f4f6"
format_stamp: "Formatted at 2024-05-07 08:06:41 on dist-test-slave-j3cx"
I20240507 08:06:41.519403 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240507 08:06:41.522693 30686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:41.523438 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240507 08:06:41.523814 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "b06daaf3dab0462a8f21abec33a5f4f6"
format_stamp: "Formatted at 2024-05-07 08:06:41 on dist-test-slave-j3cx"
I20240507 08:06:41.524109 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:41.542264 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:41.543408 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:41.548645 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:41.549014 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:41.549358 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:41.549579 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:41.598134 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:39517
I20240507 08:06:41.598232 30756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:39517 every 8 connection(s)
I20240507 08:06:41.602422 30757 heartbeater.cc:347] Connected to a master server at 127.27.100.190:45831
I20240507 08:06:41.602820 30757 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:41.603544 30757 heartbeater.cc:511] Master 127.27.100.190:45831 requested a full tablet report, sending...
I20240507 08:06:41.605297 30622 ts_manager.cc:194] Registered new tserver with Master: b06daaf3dab0462a8f21abec33a5f4f6 (127.27.100.129:39517)
I20240507 08:06:41.606107 28050 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004837491s
I20240507 08:06:41.607321 30622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51702
I20240507 08:06:41.637535 30722 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20240507 08:06:41.638926 30722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:41.650916 30767 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: Bootstrap starting.
I20240507 08:06:41.655051 30767 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:41.658931 30767 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: No bootstrap required, opened a new log
I20240507 08:06:41.659287 30767 ts_tablet_manager.cc:1367] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240507 08:06:41.661180 30767 raft_consensus.cc:348] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39517 } }
I20240507 08:06:41.661594 30767 raft_consensus.cc:374] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:41.661779 30767 raft_consensus.cc:729] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06daaf3dab0462a8f21abec33a5f4f6, State: Initialized, Role: FOLLOWER
I20240507 08:06:41.662184 30767 consensus_queue.cc:260] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39517 } }
I20240507 08:06:41.662593 30767 raft_consensus.cc:388] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:06:41.662775 30767 raft_consensus.cc:482] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:06:41.662974 30767 raft_consensus.cc:3037] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:41.667183 30767 raft_consensus.cc:504] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39517 } }
I20240507 08:06:41.667730 30767 leader_election.cc:302] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b06daaf3dab0462a8f21abec33a5f4f6; no voters: 
I20240507 08:06:41.668869 30767 leader_election.cc:288] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:06:41.669226 30769 raft_consensus.cc:2781] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:41.670819 30757 heartbeater.cc:503] Master 127.27.100.190:45831 was elected leader, sending a full tablet report...
I20240507 08:06:41.671084 30767 ts_tablet_manager.cc:1398] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.004s
I20240507 08:06:41.671393 30769 raft_consensus.cc:686] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 1 LEADER]: Becoming Leader. State: Replica: b06daaf3dab0462a8f21abec33a5f4f6, State: Running, Role: LEADER
I20240507 08:06:41.672081 30769 consensus_queue.cc:237] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39517 } }
I20240507 08:06:41.676155 30771 tablet_replica.cc:439] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: TxnStatusTablet state changed. Reason: New leader b06daaf3dab0462a8f21abec33a5f4f6. Latest consensus state: current_term: 1 leader_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" committed_config { opid_index: -1 peers { permanent_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39517 } } }
I20240507 08:06:41.676482 30770 tablet_replica.cc:439] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" committed_config { opid_index: -1 peers { permanent_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39517 } } }
I20240507 08:06:41.676863 30771 tablet_replica.cc:442] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: This TxnStatusTablet replica's current role is: LEADER
I20240507 08:06:41.677057 30770 tablet_replica.cc:442] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: This TxnStatusTablet replica's current role is: LEADER
I20240507 08:06:41.679317 30774 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
W20240507 08:06:41.679426 30622 catalog_manager.cc:5317] Ignoring report from unknown tablet 11111111111111111111111111111111
I20240507 08:06:41.679797 30774 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20240507 08:06:41.684032 30622 catalog_manager.cc:5317] Ignoring report from unknown tablet 11111111111111111111111111111111
I20240507 08:06:41.716722 28050 tablet_server.cc:178] TabletServer@127.27.100.129:0 shutting down...
I20240507 08:06:41.737931 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:41.738801 28050 tablet_replica.cc:330] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: stopping tablet replica
I20240507 08:06:41.739470 28050 raft_consensus.cc:2229] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:41.740101 28050 raft_consensus.cc:2258] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:41.752652 28050 tablet_server.cc:194] TabletServer@127.27.100.129:0 shutdown complete.
I20240507 08:06:41.763095 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:41.769966 30777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:41.770942 30778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:41.772680 30780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:41.773495 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:41.774376 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:41.774583 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:41.774732 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069201774716 us; error 0 us; skew 500 ppm
I20240507 08:06:41.775259 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:41.787585 28050 webserver.cc:460] Webserver started at http://127.27.100.129:38113/ using document root <none> and password file <none>
I20240507 08:06:41.788095 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:41.788262 28050 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240507 08:06:41.791879 28050 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.001s
I20240507 08:06:41.794684 30785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:41.795578 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240507 08:06:41.795924 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "b06daaf3dab0462a8f21abec33a5f4f6"
format_stamp: "Formatted at 2024-05-07 08:06:41 on dist-test-slave-j3cx"
I20240507 08:06:41.796216 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:41.828673 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:41.829635 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:41.837477 30794 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240507 08:06:41.850418 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240507 08:06:41.850687 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.014s	user 0.001s	sys 0.000s
I20240507 08:06:41.850968 28050 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240507 08:06:41.856707 28050 ts_tablet_manager.cc:590] Registered 1 tablets
I20240507 08:06:41.856942 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.006s	user 0.005s	sys 0.000s
I20240507 08:06:41.857049 30794 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: Bootstrap starting.
I20240507 08:06:41.890708 30794 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:06:41.891917 30794 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: Bootstrap complete.
I20240507 08:06:41.892711 30794 ts_tablet_manager.cc:1367] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: Time spent bootstrapping tablet: real 0.036s	user 0.021s	sys 0.011s
I20240507 08:06:41.895888 30794 raft_consensus.cc:348] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39517 } }
I20240507 08:06:41.896628 30794 raft_consensus.cc:729] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06daaf3dab0462a8f21abec33a5f4f6, State: Initialized, Role: FOLLOWER
I20240507 08:06:41.897554 30794 consensus_queue.cc:260] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39517 } }
I20240507 08:06:41.898438 30794 raft_consensus.cc:388] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:06:41.898841 30794 raft_consensus.cc:482] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:06:41.899282 30794 raft_consensus.cc:3037] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:06:41.908062 30794 raft_consensus.cc:504] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39517 } }
I20240507 08:06:41.908886 30794 leader_election.cc:302] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b06daaf3dab0462a8f21abec33a5f4f6; no voters: 
I20240507 08:06:41.910982 30794 leader_election.cc:288] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [CANDIDATE]: Term 2 election: Requested vote from peers 
I20240507 08:06:41.911248 30835 raft_consensus.cc:2781] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:06:41.919411 30835 raft_consensus.cc:686] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 2 LEADER]: Becoming Leader. State: Replica: b06daaf3dab0462a8f21abec33a5f4f6, State: Running, Role: LEADER
I20240507 08:06:41.920356 30835 consensus_queue.cc:237] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39517 } }
I20240507 08:06:41.929522 30794 ts_tablet_manager.cc:1398] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6: Time spent starting tablet: real 0.036s	user 0.025s	sys 0.012s
I20240507 08:06:41.933542 30837 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader b06daaf3dab0462a8f21abec33a5f4f6. Latest consensus state: current_term: 2 leader_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" committed_config { opid_index: -1 peers { permanent_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39517 } } }
I20240507 08:06:41.934185 30837 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20240507 08:06:41.933728 30838 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" committed_config { opid_index: -1 peers { permanent_uuid: "b06daaf3dab0462a8f21abec33a5f4f6" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39517 } } }
I20240507 08:06:41.934898 30838 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20240507 08:06:41.940781 30842 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240507 08:06:41.941430 30842 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240507 08:06:41.969753 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:39517
I20240507 08:06:41.969914 30862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:39517 every 8 connection(s)
I20240507 08:06:41.975473 30863 heartbeater.cc:347] Connected to a master server at 127.27.100.190:45831
I20240507 08:06:41.975987 30863 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:41.976953 30863 heartbeater.cc:511] Master 127.27.100.190:45831 requested a full tablet report, sending...
I20240507 08:06:41.978317 28050 tablet_server.cc:178] TabletServer@127.27.100.129:39517 shutting down...
I20240507 08:06:41.980300 30622 ts_manager.cc:194] Re-registered known tserver with Master: b06daaf3dab0462a8f21abec33a5f4f6 (127.27.100.129:39517)
W20240507 08:06:41.980980 30622 catalog_manager.cc:5317] Ignoring report from unknown tablet 11111111111111111111111111111111
I20240507 08:06:41.982214 30622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45506
I20240507 08:06:41.985059 30863 heartbeater.cc:503] Master 127.27.100.190:45831 was elected leader, sending a full tablet report...
W20240507 08:06:41.990602 30622 catalog_manager.cc:5317] Ignoring report from unknown tablet 11111111111111111111111111111111
I20240507 08:06:42.006757 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:42.007683 28050 tablet_replica.cc:330] stopping tablet replica
I20240507 08:06:42.008334 28050 raft_consensus.cc:2229] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 2 LEADER]: Raft consensus shutting down.
I20240507 08:06:42.008867 28050 raft_consensus.cc:2258] T 11111111111111111111111111111111 P b06daaf3dab0462a8f21abec33a5f4f6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:42.029681 28050 tablet_server.cc:194] TabletServer@127.27.100.129:39517 shutdown complete.
I20240507 08:06:42.040091 28050 master.cc:537] Master@127.27.100.190:45831 shutting down...
I20240507 08:06:42.060387 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:42.061010 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:42.061400 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P b4acf736ba1d4afb8ad1dbc913392575: stopping tablet replica
I20240507 08:06:42.079774 28050 master.cc:552] Master@127.27.100.190:45831 shutdown complete.
[       OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (812 ms)
[ RUN      ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20240507 08:06:42.112264 28050 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.100.190:35199
I20240507 08:06:42.113643 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:42.120685 30869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:42.121627 30870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:42.122495 30872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:42.123416 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:42.124158 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:42.124332 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:42.124481 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069202124464 us; error 0 us; skew 500 ppm
I20240507 08:06:42.124994 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:42.128008 28050 webserver.cc:460] Webserver started at http://127.27.100.190:33925/ using document root <none> and password file <none>
I20240507 08:06:42.128468 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:42.128633 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:42.128871 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:42.129917 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/master-0-root/instance:
uuid: "368925e5c33c472cbe5b42853f0a2cb7"
format_stamp: "Formatted at 2024-05-07 08:06:42 on dist-test-slave-j3cx"
I20240507 08:06:42.134444 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.002s
I20240507 08:06:42.137907 30877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:42.138732 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20240507 08:06:42.139063 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/master-0-root
uuid: "368925e5c33c472cbe5b42853f0a2cb7"
format_stamp: "Formatted at 2024-05-07 08:06:42 on dist-test-slave-j3cx"
I20240507 08:06:42.139436 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:42.152766 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:42.154075 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:42.202086 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.190:35199
I20240507 08:06:42.202132 30928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.190:35199 every 8 connection(s)
I20240507 08:06:42.206954 30929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:42.217051 30929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7: Bootstrap starting.
I20240507 08:06:42.221175 30929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:42.225291 30929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7: No bootstrap required, opened a new log
I20240507 08:06:42.227165 30929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368925e5c33c472cbe5b42853f0a2cb7" member_type: VOTER }
I20240507 08:06:42.227627 30929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:42.227846 30929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 368925e5c33c472cbe5b42853f0a2cb7, State: Initialized, Role: FOLLOWER
I20240507 08:06:42.228399 30929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368925e5c33c472cbe5b42853f0a2cb7" member_type: VOTER }
I20240507 08:06:42.228883 30929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:06:42.229106 30929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:06:42.229354 30929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:42.233800 30929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368925e5c33c472cbe5b42853f0a2cb7" member_type: VOTER }
I20240507 08:06:42.234313 30929 leader_election.cc:302] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 368925e5c33c472cbe5b42853f0a2cb7; no voters: 
I20240507 08:06:42.235740 30929 leader_election.cc:288] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:06:42.236111 30932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:42.237773 30932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [term 1 LEADER]: Becoming Leader. State: Replica: 368925e5c33c472cbe5b42853f0a2cb7, State: Running, Role: LEADER
I20240507 08:06:42.238570 30932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368925e5c33c472cbe5b42853f0a2cb7" member_type: VOTER }
I20240507 08:06:42.239254 30929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:06:42.242084 30934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 368925e5c33c472cbe5b42853f0a2cb7. Latest consensus state: current_term: 1 leader_uuid: "368925e5c33c472cbe5b42853f0a2cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368925e5c33c472cbe5b42853f0a2cb7" member_type: VOTER } }
I20240507 08:06:42.242168 30933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "368925e5c33c472cbe5b42853f0a2cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368925e5c33c472cbe5b42853f0a2cb7" member_type: VOTER } }
I20240507 08:06:42.242728 30934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:42.242966 30933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:42.254168 30937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:06:42.262090 30937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:06:42.264036 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:06:42.271832 30937 catalog_manager.cc:1347] Generated new cluster ID: f0892e67e0c542cd8b0bf677fc4c6b5b
I20240507 08:06:42.272140 30937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:06:42.289935 30937 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:06:42.291201 30937 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:06:42.308143 30937 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7: Generated new TSK 0
I20240507 08:06:42.308816 30937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:06:42.330883 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:42.338295 30950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:42.339203 30951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:42.340646 28050 server_base.cc:1034] running on GCE node
W20240507 08:06:42.341571 30953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:42.342545 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:42.342825 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:42.343024 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069202343007 us; error 0 us; skew 500 ppm
I20240507 08:06:42.343806 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:42.346622 28050 webserver.cc:460] Webserver started at http://127.27.100.129:35053/ using document root <none> and password file <none>
I20240507 08:06:42.347244 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:42.347486 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:42.347834 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:42.349284 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root/instance:
uuid: "c0dc3387776042fd9880aee5ec870a1a"
format_stamp: "Formatted at 2024-05-07 08:06:42 on dist-test-slave-j3cx"
I20240507 08:06:42.355216 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240507 08:06:42.359477 30958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:42.360389 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.004s	sys 0.000s
I20240507 08:06:42.360738 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "c0dc3387776042fd9880aee5ec870a1a"
format_stamp: "Formatted at 2024-05-07 08:06:42 on dist-test-slave-j3cx"
I20240507 08:06:42.361096 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:42.380180 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:42.381505 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:42.389341 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:42.389639 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:42.389936 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:42.390136 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:42.460505 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:33885
I20240507 08:06:42.460572 31028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:33885 every 8 connection(s)
I20240507 08:06:42.466182 31029 heartbeater.cc:347] Connected to a master server at 127.27.100.190:35199
I20240507 08:06:42.466255 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 08:06:42.466600 31029 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:42.467295 31029 heartbeater.cc:511] Master 127.27.100.190:35199 requested a full tablet report, sending...
I20240507 08:06:42.469065 30894 ts_manager.cc:194] Registered new tserver with Master: c0dc3387776042fd9880aee5ec870a1a (127.27.100.129:33885)
I20240507 08:06:42.470889 30894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51730
W20240507 08:06:42.476164 31031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:42.477246 31032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:42.478382 31034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:42.480592 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:42.481504 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:42.481763 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:42.481972 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069202481955 us; error 0 us; skew 500 ppm
I20240507 08:06:42.482719 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:42.485730 28050 webserver.cc:460] Webserver started at http://127.27.100.130:41967/ using document root <none> and password file <none>
I20240507 08:06:42.486338 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:42.486591 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:42.486901 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:42.488363 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-1-root/instance:
uuid: "beafdf60f46043748d566aedc8b202df"
format_stamp: "Formatted at 2024-05-07 08:06:42 on dist-test-slave-j3cx"
I20240507 08:06:42.494078 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240507 08:06:42.499537 31039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:42.500443 28050 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I20240507 08:06:42.500960 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-1-root
uuid: "beafdf60f46043748d566aedc8b202df"
format_stamp: "Formatted at 2024-05-07 08:06:42 on dist-test-slave-j3cx"
I20240507 08:06:42.501343 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:42.517174 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:42.518545 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:42.524721 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:42.525027 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:42.525331 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:42.525547 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:42.585855 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.130:33063
I20240507 08:06:42.585914 31108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.130:33063 every 8 connection(s)
I20240507 08:06:42.591970 31109 heartbeater.cc:347] Connected to a master server at 127.27.100.190:35199
I20240507 08:06:42.592428 31109 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:42.593329 31109 heartbeater.cc:511] Master 127.27.100.190:35199 requested a full tablet report, sending...
I20240507 08:06:42.595376 30894 ts_manager.cc:194] Registered new tserver with Master: beafdf60f46043748d566aedc8b202df (127.27.100.130:33063)
I20240507 08:06:42.595562 28050 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.005233824s
I20240507 08:06:42.596750 30894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51750
I20240507 08:06:42.630421 30994 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20240507 08:06:42.631649 30994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:42.642495 31119 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: Bootstrap starting.
I20240507 08:06:42.646613 31119 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:42.650741 31119 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: No bootstrap required, opened a new log
I20240507 08:06:42.651141 31119 ts_tablet_manager.cc:1367] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240507 08:06:42.653014 31119 raft_consensus.cc:348] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } }
I20240507 08:06:42.653425 31119 raft_consensus.cc:374] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:42.653646 31119 raft_consensus.cc:729] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0dc3387776042fd9880aee5ec870a1a, State: Initialized, Role: FOLLOWER
I20240507 08:06:42.654130 31119 consensus_queue.cc:260] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } }
I20240507 08:06:42.654574 31119 raft_consensus.cc:388] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:06:42.654786 31119 raft_consensus.cc:482] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:06:42.655045 31119 raft_consensus.cc:3037] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:42.659380 31119 raft_consensus.cc:504] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } }
I20240507 08:06:42.660040 31119 leader_election.cc:302] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0dc3387776042fd9880aee5ec870a1a; no voters: 
I20240507 08:06:42.661386 31119 leader_election.cc:288] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:06:42.661760 31121 raft_consensus.cc:2781] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:42.663473 31029 heartbeater.cc:503] Master 127.27.100.190:35199 was elected leader, sending a full tablet report...
I20240507 08:06:42.664212 31119 ts_tablet_manager.cc:1398] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.008s
I20240507 08:06:42.665545 31121 raft_consensus.cc:686] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 1 LEADER]: Becoming Leader. State: Replica: c0dc3387776042fd9880aee5ec870a1a, State: Running, Role: LEADER
I20240507 08:06:42.666258 31121 consensus_queue.cc:237] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } }
I20240507 08:06:42.670127 31123 tablet_replica.cc:439] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: TxnStatusTablet state changed. Reason: New leader c0dc3387776042fd9880aee5ec870a1a. Latest consensus state: current_term: 1 leader_uuid: "c0dc3387776042fd9880aee5ec870a1a" committed_config { opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } } }
I20240507 08:06:42.670707 31123 tablet_replica.cc:442] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: This TxnStatusTablet replica's current role is: LEADER
I20240507 08:06:42.671715 31122 tablet_replica.cc:439] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0dc3387776042fd9880aee5ec870a1a" committed_config { opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } } }
I20240507 08:06:42.672334 31122 tablet_replica.cc:442] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: This TxnStatusTablet replica's current role is: LEADER
W20240507 08:06:42.672535 30894 catalog_manager.cc:5317] Ignoring report from unknown tablet 11111111111111111111111111111111
I20240507 08:06:42.675554 31126 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240507 08:06:42.676028 31126 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20240507 08:06:42.677726 30894 catalog_manager.cc:5317] Ignoring report from unknown tablet 11111111111111111111111111111111
I20240507 08:06:42.728837 31130 ts_tablet_manager.cc:902] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df: Initiating tablet copy from peer c0dc3387776042fd9880aee5ec870a1a (127.27.100.129:33885)
I20240507 08:06:42.730587 31130 tablet_copy_client.cc:316] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df: tablet copy: Beginning tablet copy session from remote peer at address 127.27.100.129:33885
I20240507 08:06:42.739475 31014 tablet_copy_service.cc:140] P c0dc3387776042fd9880aee5ec870a1a: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer beafdf60f46043748d566aedc8b202df ({username='slave'} at 127.0.0.1:55060)
I20240507 08:06:42.739986 31014 tablet_copy_service.cc:161] P c0dc3387776042fd9880aee5ec870a1a: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer beafdf60f46043748d566aedc8b202df at {username='slave'} at 127.0.0.1:55060: session id = beafdf60f46043748d566aedc8b202df-11111111111111111111111111111111
I20240507 08:06:42.745227 31014 tablet_copy_source_session.cc:205] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:06:42.747771 31130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:42.757829 31130 tablet_copy_client.cc:791] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df: tablet copy: Starting download of 0 data blocks...
I20240507 08:06:42.758343 31130 tablet_copy_client.cc:655] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df: tablet copy: Starting download of 1 WAL segments...
I20240507 08:06:42.761628 31130 tablet_copy_client.cc:531] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:06:42.767313 31130 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df: Bootstrap starting.
I20240507 08:06:42.787154 31130 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:06:42.787899 31130 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df: Bootstrap complete.
I20240507 08:06:42.788450 31130 ts_tablet_manager.cc:1367] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df: Time spent bootstrapping tablet: real 0.021s	user 0.016s	sys 0.008s
I20240507 08:06:42.790361 31130 raft_consensus.cc:348] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } }
I20240507 08:06:42.790792 31130 raft_consensus.cc:729] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: beafdf60f46043748d566aedc8b202df, State: Initialized, Role: NON_PARTICIPANT
I20240507 08:06:42.791277 31130 consensus_queue.cc:260] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } }
I20240507 08:06:42.793072 31133 tablet_replica.cc:439] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } } }
I20240507 08:06:42.793251 31109 heartbeater.cc:503] Master 127.27.100.190:35199 was elected leader, sending a full tablet report...
I20240507 08:06:42.793635 31133 tablet_replica.cc:442] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20240507 08:06:42.793735 31130 ts_tablet_manager.cc:1398] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240507 08:06:42.795464 31014 tablet_copy_service.cc:342] P c0dc3387776042fd9880aee5ec870a1a: Request end of tablet copy session beafdf60f46043748d566aedc8b202df-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:55060
I20240507 08:06:42.795876 31014 tablet_copy_service.cc:433] P c0dc3387776042fd9880aee5ec870a1a: ending tablet copy session beafdf60f46043748d566aedc8b202df-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer beafdf60f46043748d566aedc8b202df
W20240507 08:06:42.796259 30894 catalog_manager.cc:5317] Ignoring report from unknown tablet 11111111111111111111111111111111
W20240507 08:06:42.800408 30894 catalog_manager.cc:5317] Ignoring report from unknown tablet 11111111111111111111111111111111
I20240507 08:06:42.822957 28050 tablet_server.cc:178] TabletServer@127.27.100.129:0 shutting down...
I20240507 08:06:42.847669 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:42.848532 28050 tablet_replica.cc:330] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: stopping tablet replica
I20240507 08:06:42.849155 28050 raft_consensus.cc:2229] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:42.849706 28050 raft_consensus.cc:2258] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:42.872844 28050 tablet_server.cc:194] TabletServer@127.27.100.129:0 shutdown complete.
I20240507 08:06:42.883258 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:42.890974 31134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:42.892334 31135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:42.893496 31137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:42.894982 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:42.896183 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:42.896447 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:42.896648 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069202896628 us; error 0 us; skew 500 ppm
I20240507 08:06:42.897342 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:42.900858 28050 webserver.cc:460] Webserver started at http://127.27.100.129:35053/ using document root <none> and password file <none>
I20240507 08:06:42.901343 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:42.901600 28050 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240507 08:06:42.905534 28050 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240507 08:06:42.908401 31142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:42.909327 28050 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240507 08:06:42.909685 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "c0dc3387776042fd9880aee5ec870a1a"
format_stamp: "Formatted at 2024-05-07 08:06:42 on dist-test-slave-j3cx"
I20240507 08:06:42.910127 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:42.923417 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:42.924536 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:42.932448 31151 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240507 08:06:42.940136 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240507 08:06:42.940476 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.009s	user 0.000s	sys 0.002s
I20240507 08:06:42.940899 28050 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240507 08:06:42.946343 31151 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: Bootstrap starting.
I20240507 08:06:42.952257 28050 ts_tablet_manager.cc:590] Registered 1 tablets
I20240507 08:06:42.952663 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.012s	user 0.004s	sys 0.001s
I20240507 08:06:42.988291 31151 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:06:42.989454 31151 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: Bootstrap complete.
I20240507 08:06:42.990244 31151 ts_tablet_manager.cc:1367] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: Time spent bootstrapping tablet: real 0.044s	user 0.027s	sys 0.012s
I20240507 08:06:42.993242 31151 raft_consensus.cc:348] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } }
I20240507 08:06:42.993925 31151 raft_consensus.cc:729] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0dc3387776042fd9880aee5ec870a1a, State: Initialized, Role: FOLLOWER
I20240507 08:06:42.994709 31151 consensus_queue.cc:260] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } }
I20240507 08:06:42.995587 31151 raft_consensus.cc:388] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:06:42.995994 31151 raft_consensus.cc:482] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:06:42.996448 31151 raft_consensus.cc:3037] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:06:43.005548 31151 raft_consensus.cc:504] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } }
I20240507 08:06:43.006418 31151 leader_election.cc:302] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0dc3387776042fd9880aee5ec870a1a; no voters: 
I20240507 08:06:43.014582 31198 raft_consensus.cc:2781] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:06:43.015879 31151 leader_election.cc:288] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [CANDIDATE]: Term 2 election: Requested vote from peers 
I20240507 08:06:43.019289 31151 ts_tablet_manager.cc:1398] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a: Time spent starting tablet: real 0.029s	user 0.015s	sys 0.013s
I20240507 08:06:43.020216 31198 raft_consensus.cc:686] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 2 LEADER]: Becoming Leader. State: Replica: c0dc3387776042fd9880aee5ec870a1a, State: Running, Role: LEADER
I20240507 08:06:43.021286 31198 consensus_queue.cc:237] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } }
I20240507 08:06:43.025071 31205 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader c0dc3387776042fd9880aee5ec870a1a. Latest consensus state: current_term: 2 leader_uuid: "c0dc3387776042fd9880aee5ec870a1a" committed_config { opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } } }
I20240507 08:06:43.025815 31205 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20240507 08:06:43.027935 31207 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c0dc3387776042fd9880aee5ec870a1a" committed_config { opid_index: -1 peers { permanent_uuid: "c0dc3387776042fd9880aee5ec870a1a" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 33885 } } }
I20240507 08:06:43.028708 31207 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20240507 08:06:43.034442 31212 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240507 08:06:43.035043 31212 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240507 08:06:43.044803 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:33885
I20240507 08:06:43.044903 31220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:33885 every 8 connection(s)
I20240507 08:06:43.050621 31221 heartbeater.cc:347] Connected to a master server at 127.27.100.190:35199
I20240507 08:06:43.051091 31221 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:43.051939 31221 heartbeater.cc:511] Master 127.27.100.190:35199 requested a full tablet report, sending...
I20240507 08:06:43.052909 28050 tablet_server.cc:178] TabletServer@127.27.100.129:33885 shutting down...
I20240507 08:06:43.056514 30894 ts_manager.cc:194] Re-registered known tserver with Master: c0dc3387776042fd9880aee5ec870a1a (127.27.100.129:33885)
W20240507 08:06:43.057207 30894 catalog_manager.cc:5317] Ignoring report from unknown tablet 11111111111111111111111111111111
I20240507 08:06:43.058941 30894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51778
I20240507 08:06:43.062286 31221 heartbeater.cc:503] Master 127.27.100.190:35199 was elected leader, sending a full tablet report...
W20240507 08:06:43.065191 30894 catalog_manager.cc:5317] Ignoring report from unknown tablet 11111111111111111111111111111111
I20240507 08:06:43.086946 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:43.087718 28050 tablet_replica.cc:330] stopping tablet replica
I20240507 08:06:43.088299 28050 raft_consensus.cc:2229] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 2 LEADER]: Raft consensus shutting down.
I20240507 08:06:43.088730 28050 raft_consensus.cc:2258] T 11111111111111111111111111111111 P c0dc3387776042fd9880aee5ec870a1a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:43.100952 28050 tablet_server.cc:194] TabletServer@127.27.100.129:33885 shutdown complete.
I20240507 08:06:43.114399 28050 tablet_server.cc:178] TabletServer@127.27.100.130:0 shutting down...
I20240507 08:06:43.136071 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:43.136831 28050 tablet_replica.cc:330] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df: stopping tablet replica
I20240507 08:06:43.137459 28050 raft_consensus.cc:2229] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20240507 08:06:43.137871 28050 raft_consensus.cc:2258] T 11111111111111111111111111111111 P beafdf60f46043748d566aedc8b202df [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20240507 08:06:43.157402 28050 tablet_server.cc:194] TabletServer@127.27.100.130:0 shutdown complete.
I20240507 08:06:43.168242 28050 master.cc:537] Master@127.27.100.190:35199 shutting down...
I20240507 08:06:43.188535 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:43.189014 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:43.189318 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P 368925e5c33c472cbe5b42853f0a2cb7: stopping tablet replica
I20240507 08:06:43.207540 28050 master.cc:552] Master@127.27.100.190:35199 shutdown complete.
[       OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (1130 ms)
[ RUN      ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20240507 08:06:43.242648 28050 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.100.190:36025
I20240507 08:06:43.243682 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:43.250276 31227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:43.254487 31228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:43.255812 31230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:43.256662 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:43.257450 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:43.257635 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:43.257788 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069203257771 us; error 0 us; skew 500 ppm
I20240507 08:06:43.258296 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:43.261196 28050 webserver.cc:460] Webserver started at http://127.27.100.190:42195/ using document root <none> and password file <none>
I20240507 08:06:43.261662 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:43.261847 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:43.262091 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:43.263212 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1715069146555183-28050-0/minicluster-data/master-0-root/instance:
uuid: "9fcd661c583d4ef3959e76f0b2c09cf9"
format_stamp: "Formatted at 2024-05-07 08:06:43 on dist-test-slave-j3cx"
I20240507 08:06:43.267740 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240507 08:06:43.271126 31235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:43.271945 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240507 08:06:43.272212 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1715069146555183-28050-0/minicluster-data/master-0-root
uuid: "9fcd661c583d4ef3959e76f0b2c09cf9"
format_stamp: "Formatted at 2024-05-07 08:06:43 on dist-test-slave-j3cx"
I20240507 08:06:43.272471 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1715069146555183-28050-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1715069146555183-28050-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1715069146555183-28050-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:43.283363 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:43.284380 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:43.334662 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.190:36025
I20240507 08:06:43.334744 31286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.190:36025 every 8 connection(s)
I20240507 08:06:43.338866 31287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:43.348304 31287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9: Bootstrap starting.
I20240507 08:06:43.352246 31287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:43.356148 31287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9: No bootstrap required, opened a new log
I20240507 08:06:43.357990 31287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd661c583d4ef3959e76f0b2c09cf9" member_type: VOTER }
I20240507 08:06:43.358353 31287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:43.358572 31287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fcd661c583d4ef3959e76f0b2c09cf9, State: Initialized, Role: FOLLOWER
I20240507 08:06:43.359133 31287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd661c583d4ef3959e76f0b2c09cf9" member_type: VOTER }
I20240507 08:06:43.359593 31287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:06:43.359807 31287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:06:43.360067 31287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:43.364255 31287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd661c583d4ef3959e76f0b2c09cf9" member_type: VOTER }
I20240507 08:06:43.364753 31287 leader_election.cc:302] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fcd661c583d4ef3959e76f0b2c09cf9; no voters: 
I20240507 08:06:43.365896 31287 leader_election.cc:288] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:06:43.366245 31290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:43.367815 31290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [term 1 LEADER]: Becoming Leader. State: Replica: 9fcd661c583d4ef3959e76f0b2c09cf9, State: Running, Role: LEADER
I20240507 08:06:43.368516 31290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd661c583d4ef3959e76f0b2c09cf9" member_type: VOTER }
I20240507 08:06:43.368881 31287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:06:43.371428 31291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fcd661c583d4ef3959e76f0b2c09cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd661c583d4ef3959e76f0b2c09cf9" member_type: VOTER } }
I20240507 08:06:43.371544 31292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fcd661c583d4ef3959e76f0b2c09cf9. Latest consensus state: current_term: 1 leader_uuid: "9fcd661c583d4ef3959e76f0b2c09cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fcd661c583d4ef3959e76f0b2c09cf9" member_type: VOTER } }
I20240507 08:06:43.372224 31291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:43.372331 31292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:43.387009 31295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:06:43.394666 31295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:06:43.395761 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:06:43.403684 31295 catalog_manager.cc:1347] Generated new cluster ID: e0f0632eee044c2bb3442809c1479006
I20240507 08:06:43.403977 31295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:06:43.419150 31295 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:06:43.420784 31295 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:06:43.441380 31295 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9: Generated new TSK 0
I20240507 08:06:43.442893 31295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:06:43.462764 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:43.470820 31308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:43.472012 31309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:43.472940 31311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:43.474413 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:43.475320 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:43.475597 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:43.475800 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069203475783 us; error 0 us; skew 500 ppm
I20240507 08:06:43.476541 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:43.479319 28050 webserver.cc:460] Webserver started at http://127.27.100.129:39167/ using document root <none> and password file <none>
I20240507 08:06:43.479950 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:43.480185 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:43.480525 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:43.481935 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1715069146555183-28050-0/minicluster-data/ts-0-root/instance:
uuid: "33ec288f5b754478b74b32da13295614"
format_stamp: "Formatted at 2024-05-07 08:06:43 on dist-test-slave-j3cx"
I20240507 08:06:43.487780 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.005s
I20240507 08:06:43.491935 31316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:43.492724 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.002s
I20240507 08:06:43.493005 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "33ec288f5b754478b74b32da13295614"
format_stamp: "Formatted at 2024-05-07 08:06:43 on dist-test-slave-j3cx"
I20240507 08:06:43.493283 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:43.522024 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:43.523048 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:43.533027 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:43.533257 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:43.533500 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:43.533660 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:43.601455 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:41055
I20240507 08:06:43.601537 31386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:41055 every 8 connection(s)
I20240507 08:06:43.606159 31387 heartbeater.cc:347] Connected to a master server at 127.27.100.190:36025
I20240507 08:06:43.606492 31387 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:43.607143 31387 heartbeater.cc:511] Master 127.27.100.190:36025 requested a full tablet report, sending...
I20240507 08:06:43.609017 31252 ts_manager.cc:194] Registered new tserver with Master: 33ec288f5b754478b74b32da13295614 (127.27.100.129:41055)
I20240507 08:06:43.609634 28050 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.00490886s
I20240507 08:06:43.610633 31252 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34524
I20240507 08:06:43.642889 31352 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20240507 08:06:43.644414 31352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:43.660037 31397 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614: Bootstrap starting.
I20240507 08:06:43.664160 31397 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:43.668181 31397 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614: No bootstrap required, opened a new log
I20240507 08:06:43.668548 31397 ts_tablet_manager.cc:1367] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240507 08:06:43.670342 31397 raft_consensus.cc:348] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "33ec288f5b754478b74b32da13295614" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 41055 } }
I20240507 08:06:43.670725 31397 raft_consensus.cc:374] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:43.670945 31397 raft_consensus.cc:729] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ec288f5b754478b74b32da13295614, State: Initialized, Role: FOLLOWER
I20240507 08:06:43.671445 31397 consensus_queue.cc:260] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "33ec288f5b754478b74b32da13295614" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 41055 } }
I20240507 08:06:43.671944 31397 raft_consensus.cc:388] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:06:43.672158 31397 raft_consensus.cc:482] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:06:43.672469 31397 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:43.676883 31397 raft_consensus.cc:504] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "33ec288f5b754478b74b32da13295614" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 41055 } }
I20240507 08:06:43.677371 31397 leader_election.cc:302] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33ec288f5b754478b74b32da13295614; no voters: 
I20240507 08:06:43.678685 31397 leader_election.cc:288] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:06:43.679029 31399 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:43.680763 31387 heartbeater.cc:503] Master 127.27.100.190:36025 was elected leader, sending a full tablet report...
I20240507 08:06:43.681723 31397 ts_tablet_manager.cc:1398] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614: Time spent starting tablet: real 0.013s	user 0.011s	sys 0.001s
I20240507 08:06:43.681751 31399 raft_consensus.cc:686] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [term 1 LEADER]: Becoming Leader. State: Replica: 33ec288f5b754478b74b32da13295614, State: Running, Role: LEADER
I20240507 08:06:43.682699 31399 consensus_queue.cc:237] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "33ec288f5b754478b74b32da13295614" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 41055 } }
I20240507 08:06:43.686190 31400 tablet_replica.cc:439] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33ec288f5b754478b74b32da13295614" committed_config { opid_index: -1 peers { permanent_uuid: "33ec288f5b754478b74b32da13295614" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 41055 } } }
I20240507 08:06:43.686756 31400 tablet_replica.cc:442] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614: This TxnStatusTablet replica's current role is: LEADER
I20240507 08:06:43.686895 31401 tablet_replica.cc:439] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614: TxnStatusTablet state changed. Reason: New leader 33ec288f5b754478b74b32da13295614. Latest consensus state: current_term: 1 leader_uuid: "33ec288f5b754478b74b32da13295614" committed_config { opid_index: -1 peers { permanent_uuid: "33ec288f5b754478b74b32da13295614" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 41055 } } }
I20240507 08:06:43.687443 31401 tablet_replica.cc:442] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614: This TxnStatusTablet replica's current role is: LEADER
W20240507 08:06:43.689584 31252 catalog_manager.cc:5317] Ignoring report from unknown tablet 11111111111111111111111111111111
I20240507 08:06:43.690364 31403 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240507 08:06:43.690738 31403 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20240507 08:06:43.694197 31252 catalog_manager.cc:5317] Ignoring report from unknown tablet 11111111111111111111111111111111
W20240507 08:06:43.743103 31340 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20240507 08:06:43.743400 31340 txn_status_manager.cc:243] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20240507 08:06:43.743739 31340 txn_status_manager.cc:205] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20240507 08:06:43.751761 31340 txn_status_manager.cc:336] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20240507 08:06:43.790844 31340 txn_status_manager.cc:243] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20240507 08:06:43.791227 31340 txn_status_manager.cc:205] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20240507 08:06:43.799916 31340 txn_status_manager.cc:336] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20240507 08:06:43.810483 31352 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:41244
W20240507 08:06:43.812667 31352 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:41244
W20240507 08:06:43.814752 31352 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:41244
W20240507 08:06:43.816606 31352 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:41244
W20240507 08:06:43.818395 31352 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:41244
W20240507 08:06:43.820201 31352 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:41244
W20240507 08:06:43.829418 31352 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:41246
W20240507 08:06:43.831147 31352 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:41246
W20240507 08:06:43.832857 31352 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:41246
W20240507 08:06:43.834440 31352 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:41246
W20240507 08:06:43.836073 31352 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:41246
W20240507 08:06:43.837666 31352 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:41246
I20240507 08:06:43.840382 28050 tablet_server.cc:178] TabletServer@127.27.100.129:0 shutting down...
I20240507 08:06:43.864620 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:43.865723 28050 tablet_replica.cc:330] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614: stopping tablet replica
I20240507 08:06:43.866422 28050 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:43.867040 28050 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 33ec288f5b754478b74b32da13295614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:43.890378 28050 tablet_server.cc:194] TabletServer@127.27.100.129:0 shutdown complete.
I20240507 08:06:43.902141 28050 master.cc:537] Master@127.27.100.190:36025 shutting down...
I20240507 08:06:43.919639 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:43.920167 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:43.920578 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9fcd661c583d4ef3959e76f0b2c09cf9: stopping tablet replica
I20240507 08:06:43.928799 28050 master.cc:552] Master@127.27.100.190:36025 shutdown complete.
[       OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (711 ms)
[ RUN      ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20240507 08:06:43.952518 28050 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.100.190:45597
I20240507 08:06:43.953565 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:43.960240 31413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:43.961390 31414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:43.962191 31416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:43.964557 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:43.965287 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:43.965498 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:43.965659 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069203965639 us; error 0 us; skew 500 ppm
I20240507 08:06:43.966156 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:43.968885 28050 webserver.cc:460] Webserver started at http://127.27.100.190:38405/ using document root <none> and password file <none>
I20240507 08:06:43.969350 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:43.969549 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:43.969807 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:43.970911 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1715069146555183-28050-0/minicluster-data/master-0-root/instance:
uuid: "fa112b7b9cf3410ca19ad586b44ef9ab"
format_stamp: "Formatted at 2024-05-07 08:06:43 on dist-test-slave-j3cx"
I20240507 08:06:43.975473 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240507 08:06:43.978978 31421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:43.979840 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.003s
I20240507 08:06:43.980113 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1715069146555183-28050-0/minicluster-data/master-0-root
uuid: "fa112b7b9cf3410ca19ad586b44ef9ab"
format_stamp: "Formatted at 2024-05-07 08:06:43 on dist-test-slave-j3cx"
I20240507 08:06:43.980531 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1715069146555183-28050-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1715069146555183-28050-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1715069146555183-28050-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:44.001003 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:44.002023 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:44.045495 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.190:45597
I20240507 08:06:44.045564 31472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.190:45597 every 8 connection(s)
I20240507 08:06:44.049177 31473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:44.058504 31473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab: Bootstrap starting.
I20240507 08:06:44.062358 31473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:44.066082 31473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab: No bootstrap required, opened a new log
I20240507 08:06:44.067888 31473 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa112b7b9cf3410ca19ad586b44ef9ab" member_type: VOTER }
I20240507 08:06:44.068250 31473 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:44.068432 31473 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa112b7b9cf3410ca19ad586b44ef9ab, State: Initialized, Role: FOLLOWER
I20240507 08:06:44.068868 31473 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa112b7b9cf3410ca19ad586b44ef9ab" member_type: VOTER }
I20240507 08:06:44.069265 31473 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:06:44.069454 31473 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:06:44.069651 31473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:44.073923 31473 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa112b7b9cf3410ca19ad586b44ef9ab" member_type: VOTER }
I20240507 08:06:44.074378 31473 leader_election.cc:302] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa112b7b9cf3410ca19ad586b44ef9ab; no voters: 
I20240507 08:06:44.075384 31473 leader_election.cc:288] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:06:44.075728 31476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:44.077133 31476 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [term 1 LEADER]: Becoming Leader. State: Replica: fa112b7b9cf3410ca19ad586b44ef9ab, State: Running, Role: LEADER
I20240507 08:06:44.077788 31476 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa112b7b9cf3410ca19ad586b44ef9ab" member_type: VOTER }
I20240507 08:06:44.078275 31473 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:06:44.080366 31477 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa112b7b9cf3410ca19ad586b44ef9ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa112b7b9cf3410ca19ad586b44ef9ab" member_type: VOTER } }
I20240507 08:06:44.080389 31478 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa112b7b9cf3410ca19ad586b44ef9ab. Latest consensus state: current_term: 1 leader_uuid: "fa112b7b9cf3410ca19ad586b44ef9ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa112b7b9cf3410ca19ad586b44ef9ab" member_type: VOTER } }
I20240507 08:06:44.081146 31477 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:44.081544 31478 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:44.084105 31481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:06:44.089447 31481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:06:44.092609 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:06:44.098022 31481 catalog_manager.cc:1347] Generated new cluster ID: c67ede5f4a214852bc03a4335e60636b
I20240507 08:06:44.098287 31481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:06:44.111860 31481 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:06:44.113031 31481 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:06:44.132369 31481 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab: Generated new TSK 0
I20240507 08:06:44.132936 31481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:06:44.159238 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:44.165755 31494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:44.166657 31495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:44.167794 31497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:44.168610 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:44.169394 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:44.169572 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:44.169737 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069204169722 us; error 0 us; skew 500 ppm
I20240507 08:06:44.170274 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:44.172617 28050 webserver.cc:460] Webserver started at http://127.27.100.129:46019/ using document root <none> and password file <none>
I20240507 08:06:44.173076 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:44.173234 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:44.173545 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:44.174593 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1715069146555183-28050-0/minicluster-data/ts-0-root/instance:
uuid: "1aafacc1f0c54d6b9674e137b5e6c7e4"
format_stamp: "Formatted at 2024-05-07 08:06:44 on dist-test-slave-j3cx"
I20240507 08:06:44.179167 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240507 08:06:44.182664 31502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:44.183418 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240507 08:06:44.183750 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "1aafacc1f0c54d6b9674e137b5e6c7e4"
format_stamp: "Formatted at 2024-05-07 08:06:44 on dist-test-slave-j3cx"
I20240507 08:06:44.184019 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:44.210216 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:44.211225 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:44.215900 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:44.216285 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:44.216692 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:44.216974 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:44.270066 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:39917
I20240507 08:06:44.270166 31572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:39917 every 8 connection(s)
I20240507 08:06:44.274351 31573 heartbeater.cc:347] Connected to a master server at 127.27.100.190:45597
I20240507 08:06:44.274727 31573 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:44.275374 31573 heartbeater.cc:511] Master 127.27.100.190:45597 requested a full tablet report, sending...
I20240507 08:06:44.277163 31438 ts_manager.cc:194] Registered new tserver with Master: 1aafacc1f0c54d6b9674e137b5e6c7e4 (127.27.100.129:39917)
I20240507 08:06:44.277978 28050 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004873551s
I20240507 08:06:44.279191 31438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43838
I20240507 08:06:44.307664 31538 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20240507 08:06:44.308758 31538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:44.319128 31583 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4: Bootstrap starting.
I20240507 08:06:44.322921 31583 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:44.326576 31583 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4: No bootstrap required, opened a new log
I20240507 08:06:44.326947 31583 ts_tablet_manager.cc:1367] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240507 08:06:44.328732 31583 raft_consensus.cc:348] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "1aafacc1f0c54d6b9674e137b5e6c7e4" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39917 } }
I20240507 08:06:44.329209 31583 raft_consensus.cc:374] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:44.329455 31583 raft_consensus.cc:729] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aafacc1f0c54d6b9674e137b5e6c7e4, State: Initialized, Role: FOLLOWER
I20240507 08:06:44.329985 31583 consensus_queue.cc:260] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "1aafacc1f0c54d6b9674e137b5e6c7e4" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39917 } }
I20240507 08:06:44.330566 31583 raft_consensus.cc:388] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:06:44.330796 31583 raft_consensus.cc:482] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:06:44.331034 31583 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:44.335738 31583 raft_consensus.cc:504] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "1aafacc1f0c54d6b9674e137b5e6c7e4" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39917 } }
I20240507 08:06:44.336287 31583 leader_election.cc:302] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aafacc1f0c54d6b9674e137b5e6c7e4; no voters: 
I20240507 08:06:44.337473 31583 leader_election.cc:288] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:06:44.337810 31585 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:44.339339 31573 heartbeater.cc:503] Master 127.27.100.190:45597 was elected leader, sending a full tablet report...
I20240507 08:06:44.339751 31583 ts_tablet_manager.cc:1398] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.008s
I20240507 08:06:44.340418 31585 raft_consensus.cc:686] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 1 LEADER]: Becoming Leader. State: Replica: 1aafacc1f0c54d6b9674e137b5e6c7e4, State: Running, Role: LEADER
I20240507 08:06:44.341099 31585 consensus_queue.cc:237] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "1aafacc1f0c54d6b9674e137b5e6c7e4" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39917 } }
I20240507 08:06:44.348489 31586 tablet_replica.cc:439] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1aafacc1f0c54d6b9674e137b5e6c7e4" committed_config { opid_index: -1 peers { permanent_uuid: "1aafacc1f0c54d6b9674e137b5e6c7e4" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39917 } } }
I20240507 08:06:44.349064 31586 tablet_replica.cc:442] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4: This TxnStatusTablet replica's current role is: LEADER
I20240507 08:06:44.350001 31587 tablet_replica.cc:439] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4: TxnStatusTablet state changed. Reason: New leader 1aafacc1f0c54d6b9674e137b5e6c7e4. Latest consensus state: current_term: 1 leader_uuid: "1aafacc1f0c54d6b9674e137b5e6c7e4" committed_config { opid_index: -1 peers { permanent_uuid: "1aafacc1f0c54d6b9674e137b5e6c7e4" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39917 } } }
W20240507 08:06:44.353603 31438 catalog_manager.cc:5317] Ignoring report from unknown tablet 11111111111111111111111111111111
I20240507 08:06:44.350807 31587 tablet_replica.cc:442] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4: This TxnStatusTablet replica's current role is: LEADER
I20240507 08:06:44.359151 31589 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240507 08:06:44.359618 31589 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20240507 08:06:44.364377 31438 catalog_manager.cc:5317] Ignoring report from unknown tablet 11111111111111111111111111111111
I20240507 08:06:44.385409 31538 tablet_service.cc:1467] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20240507 08:06:44.386513 31538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:44.396162 31583 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4: Bootstrap starting.
I20240507 08:06:44.399994 31583 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:44.403689 31583 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4: No bootstrap required, opened a new log
I20240507 08:06:44.404048 31583 ts_tablet_manager.cc:1367] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20240507 08:06:44.405761 31583 raft_consensus.cc:348] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "1aafacc1f0c54d6b9674e137b5e6c7e4" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39917 } }
I20240507 08:06:44.406159 31583 raft_consensus.cc:374] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:44.406410 31583 raft_consensus.cc:729] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aafacc1f0c54d6b9674e137b5e6c7e4, State: Initialized, Role: FOLLOWER
I20240507 08:06:44.406867 31583 consensus_queue.cc:260] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "1aafacc1f0c54d6b9674e137b5e6c7e4" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39917 } }
I20240507 08:06:44.407299 31583 raft_consensus.cc:388] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:06:44.407536 31583 raft_consensus.cc:482] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:06:44.407775 31583 raft_consensus.cc:3037] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:44.411998 31583 raft_consensus.cc:504] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "1aafacc1f0c54d6b9674e137b5e6c7e4" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39917 } }
I20240507 08:06:44.412514 31583 leader_election.cc:302] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aafacc1f0c54d6b9674e137b5e6c7e4; no voters: 
I20240507 08:06:44.412998 31583 leader_election.cc:288] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:06:44.413198 31586 raft_consensus.cc:2781] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:44.413659 31586 raft_consensus.cc:686] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 1 LEADER]: Becoming Leader. State: Replica: 1aafacc1f0c54d6b9674e137b5e6c7e4, State: Running, Role: LEADER
I20240507 08:06:44.414371 31583 ts_tablet_manager.cc:1398] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20240507 08:06:44.414294 31586 consensus_queue.cc:237] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "1aafacc1f0c54d6b9674e137b5e6c7e4" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39917 } }
W20240507 08:06:44.418916 31438 catalog_manager.cc:5317] Ignoring report from unknown tablet 22222222222222222222222222222222
W20240507 08:06:44.422456 31438 catalog_manager.cc:5317] Ignoring report from unknown tablet 22222222222222222222222222222222
I20240507 08:06:44.444901 28050 tablet_server.cc:178] TabletServer@127.27.100.129:0 shutting down...
I20240507 08:06:44.462469 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:44.463318 28050 tablet_replica.cc:330] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4: stopping tablet replica
I20240507 08:06:44.463920 28050 raft_consensus.cc:2229] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:44.464336 28050 raft_consensus.cc:2258] T 22222222222222222222222222222222 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:44.466586 28050 tablet_replica.cc:330] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4: stopping tablet replica
I20240507 08:06:44.467029 28050 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:44.467433 28050 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 1aafacc1f0c54d6b9674e137b5e6c7e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:44.477818 28050 tablet_server.cc:194] TabletServer@127.27.100.129:0 shutdown complete.
I20240507 08:06:44.488850 28050 master.cc:537] Master@127.27.100.190:45597 shutting down...
I20240507 08:06:44.502709 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:44.503237 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:44.503720 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P fa112b7b9cf3410ca19ad586b44ef9ab: stopping tablet replica
I20240507 08:06:44.521600 28050 master.cc:552] Master@127.27.100.190:45597 shutdown complete.
[       OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (592 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (3246 ms total)

[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN      ] FailedTabletsAreReplacedITest.OneReplica/0
I20240507 08:06:44.545042 28050 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.100.190:40003
I20240507 08:06:44.546046 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:44.552042 31599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:44.552824 31600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:44.553822 31602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:44.554776 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:44.555456 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:44.555688 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:44.555833 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069204555817 us; error 0 us; skew 500 ppm
I20240507 08:06:44.556363 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:44.558670 28050 webserver.cc:460] Webserver started at http://127.27.100.190:39013/ using document root <none> and password file <none>
I20240507 08:06:44.559115 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:44.559290 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:44.559561 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:44.560627 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/master-0-root/instance:
uuid: "81f2a20dc5b7475797ba7d93c5177143"
format_stamp: "Formatted at 2024-05-07 08:06:44 on dist-test-slave-j3cx"
I20240507 08:06:44.564970 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.003s
I20240507 08:06:44.568099 31607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:44.568789 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.002s
I20240507 08:06:44.569065 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/master-0-root
uuid: "81f2a20dc5b7475797ba7d93c5177143"
format_stamp: "Formatted at 2024-05-07 08:06:44 on dist-test-slave-j3cx"
I20240507 08:06:44.569334 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:44.589051 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:44.590034 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:44.623785 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.190:40003
I20240507 08:06:44.623889 31658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.190:40003 every 8 connection(s)
I20240507 08:06:44.627350 31659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:44.637347 31659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143: Bootstrap starting.
I20240507 08:06:44.641465 31659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:44.645216 31659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143: No bootstrap required, opened a new log
I20240507 08:06:44.646989 31659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81f2a20dc5b7475797ba7d93c5177143" member_type: VOTER }
I20240507 08:06:44.647368 31659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:44.647567 31659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f2a20dc5b7475797ba7d93c5177143, State: Initialized, Role: FOLLOWER
I20240507 08:06:44.648025 31659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81f2a20dc5b7475797ba7d93c5177143" member_type: VOTER }
I20240507 08:06:44.648635 31659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:06:44.648895 31659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:06:44.649112 31659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:44.653424 31659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81f2a20dc5b7475797ba7d93c5177143" member_type: VOTER }
I20240507 08:06:44.653905 31659 leader_election.cc:302] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81f2a20dc5b7475797ba7d93c5177143; no voters: 
I20240507 08:06:44.654906 31659 leader_election.cc:288] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:06:44.655216 31662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:44.656538 31662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [term 1 LEADER]: Becoming Leader. State: Replica: 81f2a20dc5b7475797ba7d93c5177143, State: Running, Role: LEADER
I20240507 08:06:44.657186 31662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81f2a20dc5b7475797ba7d93c5177143" member_type: VOTER }
I20240507 08:06:44.657716 31659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:06:44.661062 31663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81f2a20dc5b7475797ba7d93c5177143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81f2a20dc5b7475797ba7d93c5177143" member_type: VOTER } }
I20240507 08:06:44.661216 31664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81f2a20dc5b7475797ba7d93c5177143. Latest consensus state: current_term: 1 leader_uuid: "81f2a20dc5b7475797ba7d93c5177143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81f2a20dc5b7475797ba7d93c5177143" member_type: VOTER } }
I20240507 08:06:44.661994 31663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:44.662012 31664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:44.665207 31669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:06:44.672742 31669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:06:44.673687 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:06:44.680944 31669 catalog_manager.cc:1347] Generated new cluster ID: 7f1bf253f72249dbb987155f5c59250f
I20240507 08:06:44.681252 31669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:06:44.703900 31669 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:06:44.705132 31669 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:06:44.717909 31669 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143: Generated new TSK 0
I20240507 08:06:44.718542 31669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:06:44.740352 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:44.747581 31680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:44.748507 31681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:44.749538 31683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:44.751021 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:44.753530 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:44.753721 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:44.753875 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069204753858 us; error 0 us; skew 500 ppm
I20240507 08:06:44.754411 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:44.756680 28050 webserver.cc:460] Webserver started at http://127.27.100.129:37101/ using document root <none> and password file <none>
I20240507 08:06:44.757129 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:44.757297 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:44.757540 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:44.758685 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-0-root/instance:
uuid: "6dcb010d395a422ea2746e1c3fc744c0"
format_stamp: "Formatted at 2024-05-07 08:06:44 on dist-test-slave-j3cx"
I20240507 08:06:44.763031 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.003s
I20240507 08:06:44.766232 31688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:44.766927 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240507 08:06:44.767216 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "6dcb010d395a422ea2746e1c3fc744c0"
format_stamp: "Formatted at 2024-05-07 08:06:44 on dist-test-slave-j3cx"
I20240507 08:06:44.767570 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:44.786108 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:44.787083 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:44.788460 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:44.790601 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:44.790791 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:44.791026 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:44.791180 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:44.827561 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:39097
I20240507 08:06:44.827646 31750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:39097 every 8 connection(s)
I20240507 08:06:44.832044 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:44.839948 31755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:44.840881 31756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:44.841990 31758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:44.843158 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:44.843888 31751 heartbeater.cc:347] Connected to a master server at 127.27.100.190:40003
I20240507 08:06:44.844324 31751 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:44.844420 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:44.844707 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:44.844904 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069204844887 us; error 0 us; skew 500 ppm
I20240507 08:06:44.845191 31751 heartbeater.cc:511] Master 127.27.100.190:40003 requested a full tablet report, sending...
I20240507 08:06:44.845731 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:44.847404 31624 ts_manager.cc:194] Registered new tserver with Master: 6dcb010d395a422ea2746e1c3fc744c0 (127.27.100.129:39097)
I20240507 08:06:44.848472 28050 webserver.cc:460] Webserver started at http://127.27.100.130:39491/ using document root <none> and password file <none>
I20240507 08:06:44.848927 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:44.849118 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:44.849373 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:44.849503 31624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35338
I20240507 08:06:44.850543 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-1-root/instance:
uuid: "5ea2b1bb703c4255bef52caeb8768ba4"
format_stamp: "Formatted at 2024-05-07 08:06:44 on dist-test-slave-j3cx"
I20240507 08:06:44.854912 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.004s
I20240507 08:06:44.858126 31763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:44.858896 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240507 08:06:44.859184 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-1-root
uuid: "5ea2b1bb703c4255bef52caeb8768ba4"
format_stamp: "Formatted at 2024-05-07 08:06:44 on dist-test-slave-j3cx"
I20240507 08:06:44.859457 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:44.872347 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:44.873317 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:44.874636 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:44.876809 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:44.876987 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:44.877223 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:44.877360 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:44.913285 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.130:42021
I20240507 08:06:44.913388 31825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.130:42021 every 8 connection(s)
I20240507 08:06:44.917865 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:44.925989 31829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:44.928021 31830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:44.928973 31832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:44.929303 31826 heartbeater.cc:347] Connected to a master server at 127.27.100.190:40003
I20240507 08:06:44.929661 31826 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:44.929867 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:44.930532 31826 heartbeater.cc:511] Master 127.27.100.190:40003 requested a full tablet report, sending...
I20240507 08:06:44.930807 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:44.931072 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:44.931291 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069204931272 us; error 0 us; skew 500 ppm
I20240507 08:06:44.932145 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:44.932592 31624 ts_manager.cc:194] Registered new tserver with Master: 5ea2b1bb703c4255bef52caeb8768ba4 (127.27.100.130:42021)
I20240507 08:06:44.934080 31624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35348
I20240507 08:06:44.934666 28050 webserver.cc:460] Webserver started at http://127.27.100.131:35809/ using document root <none> and password file <none>
I20240507 08:06:44.935279 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:44.935556 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:44.935900 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:44.937057 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-2-root/instance:
uuid: "a7506cd8615748de8b60a852d1f999f9"
format_stamp: "Formatted at 2024-05-07 08:06:44 on dist-test-slave-j3cx"
I20240507 08:06:44.941243 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240507 08:06:44.944425 31837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:44.945196 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240507 08:06:44.945469 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-2-root
uuid: "a7506cd8615748de8b60a852d1f999f9"
format_stamp: "Formatted at 2024-05-07 08:06:44 on dist-test-slave-j3cx"
I20240507 08:06:44.945722 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:44.961120 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:44.962111 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:44.963459 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:44.965535 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:44.965729 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:44.965955 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:44.966110 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:45.002691 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.131:46675
I20240507 08:06:45.002799 31899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.131:46675 every 8 connection(s)
I20240507 08:06:45.015259 31900 heartbeater.cc:347] Connected to a master server at 127.27.100.190:40003
I20240507 08:06:45.015650 31900 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:45.016295 31900 heartbeater.cc:511] Master 127.27.100.190:40003 requested a full tablet report, sending...
I20240507 08:06:45.018234 31624 ts_manager.cc:194] Registered new tserver with Master: a7506cd8615748de8b60a852d1f999f9 (127.27.100.131:46675)
I20240507 08:06:45.018980 28050 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013454828s
I20240507 08:06:45.019910 31624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35360
I20240507 08:06:45.037339 28050 test_util.cc:268] Using random seed: -1430568698
I20240507 08:06:45.057264 31624 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35376:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20240507 08:06:45.096835 31716 tablet_service.cc:1467] Processing CreateTablet for tablet 9b503cfdc0a948d3bbc3440824a17a68 (DEFAULT_TABLE table=test-workload [id=0993947d4c914edb967b6bbb79ab3397]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:45.097981 31716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b503cfdc0a948d3bbc3440824a17a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:45.099140 31865 tablet_service.cc:1467] Processing CreateTablet for tablet 9b503cfdc0a948d3bbc3440824a17a68 (DEFAULT_TABLE table=test-workload [id=0993947d4c914edb967b6bbb79ab3397]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:45.100402 31865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b503cfdc0a948d3bbc3440824a17a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:45.102633 31791 tablet_service.cc:1467] Processing CreateTablet for tablet 9b503cfdc0a948d3bbc3440824a17a68 (DEFAULT_TABLE table=test-workload [id=0993947d4c914edb967b6bbb79ab3397]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:45.103889 31791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b503cfdc0a948d3bbc3440824a17a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:45.111258 31920 tablet_bootstrap.cc:492] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0: Bootstrap starting.
I20240507 08:06:45.115438 31920 tablet_bootstrap.cc:654] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:45.116694 31921 tablet_bootstrap.cc:492] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: Bootstrap starting.
I20240507 08:06:45.120559 31920 tablet_bootstrap.cc:492] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0: No bootstrap required, opened a new log
I20240507 08:06:45.121033 31920 ts_tablet_manager.cc:1367] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.000s
I20240507 08:06:45.122116 31923 tablet_bootstrap.cc:492] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4: Bootstrap starting.
I20240507 08:06:45.122607 31921 tablet_bootstrap.cc:654] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:45.124125 31920 raft_consensus.cc:348] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } }
I20240507 08:06:45.124909 31920 raft_consensus.cc:374] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:45.125254 31920 raft_consensus.cc:729] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dcb010d395a422ea2746e1c3fc744c0, State: Initialized, Role: FOLLOWER
I20240507 08:06:45.126735 31920 consensus_queue.cc:260] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } }
I20240507 08:06:45.128257 31923 tablet_bootstrap.cc:654] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:45.128919 31921 tablet_bootstrap.cc:492] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: No bootstrap required, opened a new log
I20240507 08:06:45.128988 31751 heartbeater.cc:503] Master 127.27.100.190:40003 was elected leader, sending a full tablet report...
I20240507 08:06:45.129442 31921 ts_tablet_manager.cc:1367] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: Time spent bootstrapping tablet: real 0.013s	user 0.010s	sys 0.000s
I20240507 08:06:45.129437 31920 ts_tablet_manager.cc:1398] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20240507 08:06:45.131929 31921 raft_consensus.cc:348] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } }
I20240507 08:06:45.132472 31921 raft_consensus.cc:374] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:45.132758 31921 raft_consensus.cc:729] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7506cd8615748de8b60a852d1f999f9, State: Initialized, Role: FOLLOWER
I20240507 08:06:45.133453 31921 consensus_queue.cc:260] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } }
I20240507 08:06:45.134763 31923 tablet_bootstrap.cc:492] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4: No bootstrap required, opened a new log
I20240507 08:06:45.135275 31923 ts_tablet_manager.cc:1367] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4: Time spent bootstrapping tablet: real 0.013s	user 0.010s	sys 0.000s
I20240507 08:06:45.135505 31900 heartbeater.cc:503] Master 127.27.100.190:40003 was elected leader, sending a full tablet report...
I20240507 08:06:45.136045 31921 ts_tablet_manager.cc:1398] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.004s
I20240507 08:06:45.138134 31923 raft_consensus.cc:348] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } }
I20240507 08:06:45.138826 31923 raft_consensus.cc:374] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:45.139123 31923 raft_consensus.cc:729] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea2b1bb703c4255bef52caeb8768ba4, State: Initialized, Role: FOLLOWER
I20240507 08:06:45.139931 31923 consensus_queue.cc:260] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } }
I20240507 08:06:45.142390 31826 heartbeater.cc:503] Master 127.27.100.190:40003 was elected leader, sending a full tablet report...
I20240507 08:06:45.142948 31923 ts_tablet_manager.cc:1398] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20240507 08:06:45.167059 31925 raft_consensus.cc:482] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:45.167567 31925 raft_consensus.cc:504] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } }
I20240507 08:06:45.169762 31925 leader_election.cc:288] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7506cd8615748de8b60a852d1f999f9 (127.27.100.131:46675), 5ea2b1bb703c4255bef52caeb8768ba4 (127.27.100.130:42021)
I20240507 08:06:45.182327 31875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b503cfdc0a948d3bbc3440824a17a68" candidate_uuid: "6dcb010d395a422ea2746e1c3fc744c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7506cd8615748de8b60a852d1f999f9" is_pre_election: true
I20240507 08:06:45.182735 31801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b503cfdc0a948d3bbc3440824a17a68" candidate_uuid: "6dcb010d395a422ea2746e1c3fc744c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" is_pre_election: true
I20240507 08:06:45.183140 31875 raft_consensus.cc:2454] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dcb010d395a422ea2746e1c3fc744c0 in term 0.
I20240507 08:06:45.183461 31801 raft_consensus.cc:2454] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dcb010d395a422ea2746e1c3fc744c0 in term 0.
I20240507 08:06:45.184345 31691 leader_election.cc:302] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dcb010d395a422ea2746e1c3fc744c0, a7506cd8615748de8b60a852d1f999f9; no voters: 
I20240507 08:06:45.185184 31925 raft_consensus.cc:2781] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:06:45.185458 31925 raft_consensus.cc:482] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:45.185672 31925 raft_consensus.cc:3037] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:45.190121 31925 raft_consensus.cc:504] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } }
I20240507 08:06:45.191777 31925 leader_election.cc:288] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [CANDIDATE]: Term 1 election: Requested vote from peers a7506cd8615748de8b60a852d1f999f9 (127.27.100.131:46675), 5ea2b1bb703c4255bef52caeb8768ba4 (127.27.100.130:42021)
I20240507 08:06:45.192374 31875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b503cfdc0a948d3bbc3440824a17a68" candidate_uuid: "6dcb010d395a422ea2746e1c3fc744c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7506cd8615748de8b60a852d1f999f9"
I20240507 08:06:45.192811 31875 raft_consensus.cc:3037] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:45.192735 31801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b503cfdc0a948d3bbc3440824a17a68" candidate_uuid: "6dcb010d395a422ea2746e1c3fc744c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea2b1bb703c4255bef52caeb8768ba4"
I20240507 08:06:45.193320 31801 raft_consensus.cc:3037] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:45.197194 31875 raft_consensus.cc:2454] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dcb010d395a422ea2746e1c3fc744c0 in term 1.
I20240507 08:06:45.197661 31801 raft_consensus.cc:2454] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dcb010d395a422ea2746e1c3fc744c0 in term 1.
I20240507 08:06:45.198048 31691 leader_election.cc:302] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dcb010d395a422ea2746e1c3fc744c0, a7506cd8615748de8b60a852d1f999f9; no voters: 
I20240507 08:06:45.198750 31925 raft_consensus.cc:2781] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:45.199640 31925 raft_consensus.cc:686] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 1 LEADER]: Becoming Leader. State: Replica: 6dcb010d395a422ea2746e1c3fc744c0, State: Running, Role: LEADER
I20240507 08:06:45.200268 31925 consensus_queue.cc:237] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } }
I20240507 08:06:45.206494 31623 catalog_manager.cc:5484] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dcb010d395a422ea2746e1c3fc744c0 (127.27.100.129). New cstate: current_term: 1 leader_uuid: "6dcb010d395a422ea2746e1c3fc744c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } } }
I20240507 08:06:45.439062 31791 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:45.439214 31716 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240507 08:06:45.444077 31865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 81f2a20dc5b7475797ba7d93c5177143 | 127.27.100.190:40003 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 5ea2b1bb703c4255bef52caeb8768ba4 | 127.27.100.130:42021 | HEALTHY | <none>   |       0        |       0
 6dcb010d395a422ea2746e1c3fc744c0 | 127.27.100.129:39097 | HEALTHY | <none>   |       1        |       0
 a7506cd8615748de8b60a852d1f999f9 | 127.27.100.131:46675 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet 9b503cfdc0a948d3bbc3440824a17a68 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
  6dcb010d395a422ea2746e1c3fc744c0 (127.27.100.129:39097): RUNNING [LEADER]
  a7506cd8615748de8b60a852d1f999f9 (127.27.100.131:46675): RUNNING
  5ea2b1bb703c4255bef52caeb8768ba4 (127.27.100.130:42021): RUNNING
All reported replicas are:
  A = 6dcb010d395a422ea2746e1c3fc744c0
  B = a7506cd8615748de8b60a852d1f999f9
  C = 5ea2b1bb703c4255bef52caeb8768ba4
The consensus matrix is:
 Config source |   Replicas   | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A*  B   C    |              |              | Yes
 A             | A*  B   C    | 1            | -1           | Yes
 B             | A   B   C    | 1            | -1           | Yes
 C             | A   B   C    | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |       Status       | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | CONSENSUS_MISMATCH | 1             | 0       | 0          | 0                | 1

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20240507 08:06:45.663892 28050 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20240507 08:06:45.727296 31958 consensus_queue.cc:1035] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:06:45.738806 31961 consensus_queue.cc:1035] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:06:46.023599 31716 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240507 08:06:46.023846 31791 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:46.031390 31865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 81f2a20dc5b7475797ba7d93c5177143 | 127.27.100.190:40003 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 5ea2b1bb703c4255bef52caeb8768ba4 | 127.27.100.130:42021 | HEALTHY | <none>   |       0        |       0
 6dcb010d395a422ea2746e1c3fc744c0 | 127.27.100.129:39097 | HEALTHY | <none>   |       1        |       0
 a7506cd8615748de8b60a852d1f999f9 | 127.27.100.131:46675 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
     Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240507 08:06:46.215160 28050 log_verifier.cc:126] Checking tablet 9b503cfdc0a948d3bbc3440824a17a68
I20240507 08:06:46.280709 28050 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 9b503cfdc0a948d3bbc3440824a17a68
I20240507 08:06:46.281313 28050 tablet_replica.cc:330] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: stopping tablet replica
I20240507 08:06:46.281837 28050 raft_consensus.cc:2229] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:46.282289 28050 raft_consensus.cc:2258] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:46.481264 31791 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:46.483580 31716 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240507 08:06:46.485085 31865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 81f2a20dc5b7475797ba7d93c5177143 | 127.27.100.190:40003 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 5ea2b1bb703c4255bef52caeb8768ba4 | 127.27.100.130:42021 | HEALTHY | <none>   |       0        |       0
 6dcb010d395a422ea2746e1c3fc744c0 | 127.27.100.129:39097 | HEALTHY | <none>   |       1        |       0
 a7506cd8615748de8b60a852d1f999f9 | 127.27.100.131:46675 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet 9b503cfdc0a948d3bbc3440824a17a68 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  6dcb010d395a422ea2746e1c3fc744c0 (127.27.100.129:39097): RUNNING [LEADER]
  a7506cd8615748de8b60a852d1f999f9 (127.27.100.131:46675): not running
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
  5ea2b1bb703c4255bef52caeb8768ba4 (127.27.100.130:42021): RUNNING
All reported replicas are:
  A = 6dcb010d395a422ea2746e1c3fc744c0
  B = a7506cd8615748de8b60a852d1f999f9
  C = 5ea2b1bb703c4255bef52caeb8768ba4
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A*  B   C              |              |              | Yes
 A             | A*  B   C              | 1            | -1           | Yes
 B             | [config not available] |              |              | 
 C             | A*  B   C              | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20240507 08:06:46.677866 28050 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20240507 08:06:46.800136 31691 consensus_peers.cc:485] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 -> Peer a7506cd8615748de8b60a852d1f999f9 (127.27.100.131:46675): Couldn't send request to peer a7506cd8615748de8b60a852d1f999f9. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:46.801090 31961 raft_consensus.cc:998] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0: Attempting to remove follower a7506cd8615748de8b60a852d1f999f9 from the Raft config. Reason: The tablet replica hosted on peer a7506cd8615748de8b60a852d1f999f9 has failed
I20240507 08:06:46.802656 31961 consensus_queue.cc:237] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } }
I20240507 08:06:46.806862 31801 raft_consensus.cc:1261] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4 [term 1 FOLLOWER]: Refusing update from remote peer 6dcb010d395a422ea2746e1c3fc744c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:06:46.807963 32016 consensus_queue.cc:1035] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:06:46.813714 31961 raft_consensus.cc:2932] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a7506cd8615748de8b60a852d1f999f9 (127.27.100.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } } }
I20240507 08:06:46.815196 31801 raft_consensus.cc:2932] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a7506cd8615748de8b60a852d1f999f9 (127.27.100.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } } }
I20240507 08:06:46.826001 31622 catalog_manager.cc:5484] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 reported cstate change: config changed from index -1 to 2, VOTER a7506cd8615748de8b60a852d1f999f9 (127.27.100.131) evicted. New cstate: current_term: 1 leader_uuid: "6dcb010d395a422ea2746e1c3fc744c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } } }
I20240507 08:06:46.842447 31865 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b503cfdc0a948d3bbc3440824a17a68 with delete_type TABLET_DATA_TOMBSTONED (TS a7506cd8615748de8b60a852d1f999f9 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:55224
I20240507 08:06:46.847088 31726 consensus_queue.cc:237] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } }
I20240507 08:06:46.858140 32030 ts_tablet_manager.cc:1875] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240507 08:06:46.859849 31691 consensus_peers.cc:485] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 -> Peer a7506cd8615748de8b60a852d1f999f9 (127.27.100.131:46675): Couldn't send request to peer a7506cd8615748de8b60a852d1f999f9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:46.862810 31801 raft_consensus.cc:1261] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4 [term 1 FOLLOWER]: Refusing update from remote peer 6dcb010d395a422ea2746e1c3fc744c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:06:46.864351 32016 consensus_queue.cc:1035] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:46.870759 31961 raft_consensus.cc:2932] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a7506cd8615748de8b60a852d1f999f9 (127.27.100.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } } }
I20240507 08:06:46.875406 32030 ts_tablet_manager.cc:1888] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240507 08:06:46.875860 32030 log.cc:1196] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: Deleting WAL directory at /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1715069146555183-28050-0/minicluster-data/ts-2-root/wals/9b503cfdc0a948d3bbc3440824a17a68
I20240507 08:06:46.877454 31610 catalog_manager.cc:4830] TS a7506cd8615748de8b60a852d1f999f9 (127.27.100.131:46675): tablet 9b503cfdc0a948d3bbc3440824a17a68 (table test-workload [id=0993947d4c914edb967b6bbb79ab3397]) successfully deleted
I20240507 08:06:46.881274 31801 raft_consensus.cc:2932] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a7506cd8615748de8b60a852d1f999f9 (127.27.100.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } } }
I20240507 08:06:46.884419 31609 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9b503cfdc0a948d3bbc3440824a17a68 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240507 08:06:46.888461 31622 catalog_manager.cc:5484] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 reported cstate change: config changed from index 2 to 3, VOTER a7506cd8615748de8b60a852d1f999f9 (127.27.100.131) added. New cstate: current_term: 1 leader_uuid: "6dcb010d395a422ea2746e1c3fc744c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } } }
I20240507 08:06:47.076705 31791 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:47.081439 31716 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240507 08:06:47.082760 31865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 81f2a20dc5b7475797ba7d93c5177143 | 127.27.100.190:40003 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 5ea2b1bb703c4255bef52caeb8768ba4 | 127.27.100.130:42021 | HEALTHY | <none>   |       0        |       0
 6dcb010d395a422ea2746e1c3fc744c0 | 127.27.100.129:39097 | HEALTHY | <none>   |       1        |       0
 a7506cd8615748de8b60a852d1f999f9 | 127.27.100.131:46675 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet 9b503cfdc0a948d3bbc3440824a17a68 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  6dcb010d395a422ea2746e1c3fc744c0 (127.27.100.129:39097): RUNNING [LEADER]
  5ea2b1bb703c4255bef52caeb8768ba4 (127.27.100.130:42021): RUNNING
  a7506cd8615748de8b60a852d1f999f9 (127.27.100.131:46675): not running
    State:       FAILED
    Data state:  TABLET_DATA_TOMBSTONED
    Last status: Deleted tablet blocks from disk
All reported replicas are:
  A = 6dcb010d395a422ea2746e1c3fc744c0
  B = 5ea2b1bb703c4255bef52caeb8768ba4
  C = a7506cd8615748de8b60a852d1f999f9
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A*  B   C              |              |              | Yes
 A             | A*  B   C              | 1            | 3            | Yes
 B             | A*  B   C              | 1            | 3            | Yes
 C             | [config not available] |              |              | 

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20240507 08:06:47.281452 28050 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20240507 08:06:47.313048 32052 ts_tablet_manager.cc:902] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: Initiating tablet copy from peer 6dcb010d395a422ea2746e1c3fc744c0 (127.27.100.129:39097)
I20240507 08:06:47.314388 32052 tablet_copy_client.cc:316] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: tablet copy: Beginning tablet copy session from remote peer at address 127.27.100.129:39097
I20240507 08:06:47.322600 31736 tablet_copy_service.cc:140] P 6dcb010d395a422ea2746e1c3fc744c0: Received BeginTabletCopySession request for tablet 9b503cfdc0a948d3bbc3440824a17a68 from peer a7506cd8615748de8b60a852d1f999f9 ({username='slave'} at 127.0.0.1:36660)
I20240507 08:06:47.323081 31736 tablet_copy_service.cc:161] P 6dcb010d395a422ea2746e1c3fc744c0: Beginning new tablet copy session on tablet 9b503cfdc0a948d3bbc3440824a17a68 from peer a7506cd8615748de8b60a852d1f999f9 at {username='slave'} at 127.0.0.1:36660: session id = a7506cd8615748de8b60a852d1f999f9-9b503cfdc0a948d3bbc3440824a17a68
I20240507 08:06:47.328064 31736 tablet_copy_source_session.cc:205] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:06:47.330078 32052 ts_tablet_manager.cc:1875] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240507 08:06:47.339907 32052 ts_tablet_manager.cc:1888] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20240507 08:06:47.340504 32052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b503cfdc0a948d3bbc3440824a17a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:47.345395 32052 tablet_copy_client.cc:791] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: tablet copy: Starting download of 0 data blocks...
I20240507 08:06:47.345844 32052 tablet_copy_client.cc:655] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: tablet copy: Starting download of 1 WAL segments...
I20240507 08:06:47.348839 32052 tablet_copy_client.cc:531] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:06:47.354189 32052 tablet_bootstrap.cc:492] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: Bootstrap starting.
I20240507 08:06:47.368925 32052 tablet_bootstrap.cc:492] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:06:47.369616 32052 tablet_bootstrap.cc:492] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: Bootstrap complete.
I20240507 08:06:47.370314 32052 ts_tablet_manager.cc:1367] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: Time spent bootstrapping tablet: real 0.016s	user 0.010s	sys 0.005s
I20240507 08:06:47.372412 32052 raft_consensus.cc:348] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } }
I20240507 08:06:47.372931 32052 raft_consensus.cc:729] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7506cd8615748de8b60a852d1f999f9, State: Initialized, Role: FOLLOWER
I20240507 08:06:47.373481 32052 consensus_queue.cc:260] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6dcb010d395a422ea2746e1c3fc744c0" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 39097 } } peers { permanent_uuid: "5ea2b1bb703c4255bef52caeb8768ba4" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 42021 } } peers { permanent_uuid: "a7506cd8615748de8b60a852d1f999f9" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 46675 } }
I20240507 08:06:47.375464 32052 ts_tablet_manager.cc:1398] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: Time spent starting tablet: real 0.005s	user 0.007s	sys 0.000s
I20240507 08:06:47.377280 31736 tablet_copy_service.cc:342] P 6dcb010d395a422ea2746e1c3fc744c0: Request end of tablet copy session a7506cd8615748de8b60a852d1f999f9-9b503cfdc0a948d3bbc3440824a17a68 received from {username='slave'} at 127.0.0.1:36660
I20240507 08:06:47.377663 31736 tablet_copy_service.cc:433] P 6dcb010d395a422ea2746e1c3fc744c0: ending tablet copy session a7506cd8615748de8b60a852d1f999f9-9b503cfdc0a948d3bbc3440824a17a68 on tablet 9b503cfdc0a948d3bbc3440824a17a68 with peer a7506cd8615748de8b60a852d1f999f9
I20240507 08:06:47.763217 31791 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:47.765667 31716 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240507 08:06:47.769629 31865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:47.905768 31875 raft_consensus.cc:1203] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 81f2a20dc5b7475797ba7d93c5177143 | 127.27.100.190:40003 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 5ea2b1bb703c4255bef52caeb8768ba4 | 127.27.100.130:42021 | HEALTHY | <none>   |       0        |       0
 6dcb010d395a422ea2746e1c3fc744c0 | 127.27.100.129:39097 | HEALTHY | <none>   |       1        |       0
 a7506cd8615748de8b60a852d1f999f9 | 127.27.100.131:46675 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
Tablet 9b503cfdc0a948d3bbc3440824a17a68 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
  6dcb010d395a422ea2746e1c3fc744c0 (127.27.100.129:39097): RUNNING [LEADER]
  5ea2b1bb703c4255bef52caeb8768ba4 (127.27.100.130:42021): RUNNING
  a7506cd8615748de8b60a852d1f999f9 (127.27.100.131:46675): RUNNING
All reported replicas are:
  A = 6dcb010d395a422ea2746e1c3fc744c0
  B = 5ea2b1bb703c4255bef52caeb8768ba4
  C = a7506cd8615748de8b60a852d1f999f9
The consensus matrix is:
 Config source |   Replicas   | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A*  B   C    |              |              | Yes
 A             | A*  B   C    | 1            | 3            | Yes
 B             | A*  B   C    | 1            | 3            | Yes
 C             | A   B   C    | 1            | 3            | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |       Status       | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | CONSENSUS_MISMATCH | 1             | 0       | 0          | 0                | 1

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20240507 08:06:47.963413 28050 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20240507 08:06:48.506428 31716 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240507 08:06:48.516494 31865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:48.520783 31791 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 81f2a20dc5b7475797ba7d93c5177143 | 127.27.100.190:40003 | HEALTHY

Unusual flags for Master:
                   Flag                   | Value |         Tags          |         Master
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 5ea2b1bb703c4255bef52caeb8768ba4 | 127.27.100.130:42021 | HEALTHY | <none>   |       0        |       0
 6dcb010d395a422ea2746e1c3fc744c0 | 127.27.100.129:39097 | HEALTHY | <none>   |       1        |       0
 a7506cd8615748de8b60a852d1f999f9 | 127.27.100.131:46675 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                   Flag                   | Value |         Tags          |      Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
 raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports         | true  | unsafe                | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system_unsync                                               | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
     Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240507 08:06:48.707523 28050 log_verifier.cc:126] Checking tablet 9b503cfdc0a948d3bbc3440824a17a68
I20240507 08:06:48.762559 28050 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9b503cfdc0a948d3bbc3440824a17a68
I20240507 08:06:48.768455 28050 tablet_server.cc:178] TabletServer@127.27.100.129:0 shutting down...
I20240507 08:06:48.788292 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:48.788900 28050 tablet_replica.cc:330] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0: stopping tablet replica
I20240507 08:06:48.789438 28050 raft_consensus.cc:2229] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:48.790319 28050 raft_consensus.cc:2258] T 9b503cfdc0a948d3bbc3440824a17a68 P 6dcb010d395a422ea2746e1c3fc744c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:48.809955 28050 tablet_server.cc:194] TabletServer@127.27.100.129:0 shutdown complete.
I20240507 08:06:48.820449 28050 tablet_server.cc:178] TabletServer@127.27.100.130:0 shutting down...
I20240507 08:06:48.840384 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:48.841097 28050 tablet_replica.cc:330] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4: stopping tablet replica
I20240507 08:06:48.841675 28050 raft_consensus.cc:2229] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:48.842202 28050 raft_consensus.cc:2258] T 9b503cfdc0a948d3bbc3440824a17a68 P 5ea2b1bb703c4255bef52caeb8768ba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:48.861493 28050 tablet_server.cc:194] TabletServer@127.27.100.130:0 shutdown complete.
I20240507 08:06:48.871351 28050 tablet_server.cc:178] TabletServer@127.27.100.131:0 shutting down...
I20240507 08:06:48.893256 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:48.893884 28050 tablet_replica.cc:330] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9: stopping tablet replica
I20240507 08:06:48.894399 28050 raft_consensus.cc:2229] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:48.894790 28050 raft_consensus.cc:2258] T 9b503cfdc0a948d3bbc3440824a17a68 P a7506cd8615748de8b60a852d1f999f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:48.914258 28050 tablet_server.cc:194] TabletServer@127.27.100.131:0 shutdown complete.
I20240507 08:06:48.924163 28050 master.cc:537] Master@127.27.100.190:40003 shutting down...
I20240507 08:06:48.941172 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:48.941738 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:48.942075 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P 81f2a20dc5b7475797ba7d93c5177143: stopping tablet replica
I20240507 08:06:48.959820 28050 master.cc:552] Master@127.27.100.190:40003 shutdown complete.
[       OK ] FailedTabletsAreReplacedITest.OneReplica/0 (4445 ms)
[ RUN      ] FailedTabletsAreReplacedITest.OneReplica/1
I20240507 08:06:48.989840 28050 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.100.190:42731
I20240507 08:06:48.990826 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:48.997229 32116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:48.998034 32117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:48.999110 32119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:48.999994 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:49.000706 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:49.000875 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:49.001031 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069209001013 us; error 0 us; skew 500 ppm
I20240507 08:06:49.001544 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:49.003996 28050 webserver.cc:460] Webserver started at http://127.27.100.190:44603/ using document root <none> and password file <none>
I20240507 08:06:49.004441 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:49.004611 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:49.004855 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:49.005982 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/master-0-root/instance:
uuid: "a204481d2a5a4d8e9b0d8e035546ad16"
format_stamp: "Formatted at 2024-05-07 08:06:49 on dist-test-slave-j3cx"
I20240507 08:06:49.010233 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240507 08:06:49.013377 32124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:49.014124 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240507 08:06:49.014391 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/master-0-root
uuid: "a204481d2a5a4d8e9b0d8e035546ad16"
format_stamp: "Formatted at 2024-05-07 08:06:49 on dist-test-slave-j3cx"
I20240507 08:06:49.014659 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:49.048311 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:49.049502 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:49.085618 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.190:42731
I20240507 08:06:49.085708 32175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.190:42731 every 8 connection(s)
I20240507 08:06:49.089445 32176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:49.099653 32176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16: Bootstrap starting.
I20240507 08:06:49.103966 32176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:49.108006 32176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16: No bootstrap required, opened a new log
I20240507 08:06:49.109987 32176 raft_consensus.cc:348] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a204481d2a5a4d8e9b0d8e035546ad16" member_type: VOTER }
I20240507 08:06:49.110433 32176 raft_consensus.cc:374] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:49.110663 32176 raft_consensus.cc:729] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a204481d2a5a4d8e9b0d8e035546ad16, State: Initialized, Role: FOLLOWER
I20240507 08:06:49.111204 32176 consensus_queue.cc:260] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a204481d2a5a4d8e9b0d8e035546ad16" member_type: VOTER }
I20240507 08:06:49.111660 32176 raft_consensus.cc:388] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:06:49.111867 32176 raft_consensus.cc:482] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:06:49.112108 32176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:49.116590 32176 raft_consensus.cc:504] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a204481d2a5a4d8e9b0d8e035546ad16" member_type: VOTER }
I20240507 08:06:49.117102 32176 leader_election.cc:302] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a204481d2a5a4d8e9b0d8e035546ad16; no voters: 
I20240507 08:06:49.118199 32176 leader_election.cc:288] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240507 08:06:49.118578 32179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:49.119845 32179 raft_consensus.cc:686] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [term 1 LEADER]: Becoming Leader. State: Replica: a204481d2a5a4d8e9b0d8e035546ad16, State: Running, Role: LEADER
I20240507 08:06:49.120471 32179 consensus_queue.cc:237] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a204481d2a5a4d8e9b0d8e035546ad16" member_type: VOTER }
I20240507 08:06:49.120981 32176 sys_catalog.cc:564] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:06:49.123613 32180 sys_catalog.cc:455] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a204481d2a5a4d8e9b0d8e035546ad16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a204481d2a5a4d8e9b0d8e035546ad16" member_type: VOTER } }
I20240507 08:06:49.123515 32181 sys_catalog.cc:455] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a204481d2a5a4d8e9b0d8e035546ad16. Latest consensus state: current_term: 1 leader_uuid: "a204481d2a5a4d8e9b0d8e035546ad16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a204481d2a5a4d8e9b0d8e035546ad16" member_type: VOTER } }
I20240507 08:06:49.124285 32180 sys_catalog.cc:458] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:49.124373 32181 sys_catalog.cc:458] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [sys.catalog]: This master's current role is: LEADER
I20240507 08:06:49.131186 32184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:06:49.135928 32184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:06:49.136898 28050 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:06:49.144011 32184 catalog_manager.cc:1347] Generated new cluster ID: 28b353bdf5214567b4cd1a5793914e3b
I20240507 08:06:49.144251 32184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:06:49.163596 32184 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:06:49.164773 32184 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:06:49.177682 32184 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16: Generated new TSK 0
I20240507 08:06:49.178246 32184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:06:49.203639 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:49.210664 32197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:49.211951 32198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:49.212929 32200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:49.213894 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:49.214615 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:49.214833 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:49.215000 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069209214982 us; error 0 us; skew 500 ppm
I20240507 08:06:49.215610 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:49.217885 28050 webserver.cc:460] Webserver started at http://127.27.100.129:38609/ using document root <none> and password file <none>
I20240507 08:06:49.218345 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:49.218523 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:49.218773 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:49.219857 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-0-root/instance:
uuid: "cb033d0ab28c460e94b755526f99df7c"
format_stamp: "Formatted at 2024-05-07 08:06:49 on dist-test-slave-j3cx"
I20240507 08:06:49.224216 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240507 08:06:49.227416 32205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:49.228201 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240507 08:06:49.228488 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-0-root
uuid: "cb033d0ab28c460e94b755526f99df7c"
format_stamp: "Formatted at 2024-05-07 08:06:49 on dist-test-slave-j3cx"
I20240507 08:06:49.228782 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:49.244758 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:49.245919 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:49.247375 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:49.249636 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:49.249851 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:49.250083 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:49.250239 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:49.289999 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.129:36785
I20240507 08:06:49.290076 32267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.129:36785 every 8 connection(s)
I20240507 08:06:49.294883 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:49.303393 32272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:49.304484 32273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:49.306931 32275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:49.307693 32268 heartbeater.cc:347] Connected to a master server at 127.27.100.190:42731
I20240507 08:06:49.308120 32268 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:49.308815 32268 heartbeater.cc:511] Master 127.27.100.190:42731 requested a full tablet report, sending...
I20240507 08:06:49.309711 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:49.310498 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:49.310703 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:49.310756 32141 ts_manager.cc:194] Registered new tserver with Master: cb033d0ab28c460e94b755526f99df7c (127.27.100.129:36785)
I20240507 08:06:49.310882 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069209310867 us; error 0 us; skew 500 ppm
I20240507 08:06:49.311631 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:49.312618 32141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37708
I20240507 08:06:49.314191 28050 webserver.cc:460] Webserver started at http://127.27.100.130:33697/ using document root <none> and password file <none>
I20240507 08:06:49.314833 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:49.315074 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:49.315325 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:49.316411 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-1-root/instance:
uuid: "20ec0da746b2424dbd6c481df2d336b8"
format_stamp: "Formatted at 2024-05-07 08:06:49 on dist-test-slave-j3cx"
I20240507 08:06:49.320402 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240507 08:06:49.323611 32280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:49.324347 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240507 08:06:49.324627 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-1-root
uuid: "20ec0da746b2424dbd6c481df2d336b8"
format_stamp: "Formatted at 2024-05-07 08:06:49 on dist-test-slave-j3cx"
I20240507 08:06:49.324908 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:49.338966 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:49.340167 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:49.341547 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:49.343701 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:49.343896 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:49.344116 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:49.344269 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:49.381470 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.130:44247
I20240507 08:06:49.381567 32342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.130:44247 every 8 connection(s)
I20240507 08:06:49.386073 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:49.394354 32346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:49.396322 32347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:49.397365 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:49.397969 32343 heartbeater.cc:347] Connected to a master server at 127.27.100.190:42731
I20240507 08:06:49.398321 32343 heartbeater.cc:464] Registering TS with master...
W20240507 08:06:49.398370 32349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:49.399118 32343 heartbeater.cc:511] Master 127.27.100.190:42731 requested a full tablet report, sending...
I20240507 08:06:49.399277 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:49.399575 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:49.399739 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069209399720 us; error 0 us; skew 500 ppm
I20240507 08:06:49.400321 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:49.400979 32141 ts_manager.cc:194] Registered new tserver with Master: 20ec0da746b2424dbd6c481df2d336b8 (127.27.100.130:44247)
I20240507 08:06:49.402489 32141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37718
I20240507 08:06:49.402995 28050 webserver.cc:460] Webserver started at http://127.27.100.131:44373/ using document root <none> and password file <none>
I20240507 08:06:49.403551 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:49.403802 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:49.404064 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:49.405211 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-2-root/instance:
uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6"
format_stamp: "Formatted at 2024-05-07 08:06:49 on dist-test-slave-j3cx"
I20240507 08:06:49.409654 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240507 08:06:49.412925 32354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:49.413630 28050 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240507 08:06:49.413928 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-2-root
uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6"
format_stamp: "Formatted at 2024-05-07 08:06:49 on dist-test-slave-j3cx"
I20240507 08:06:49.414212 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:49.454499 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:49.455704 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:49.457156 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:49.459363 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:49.459586 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:49.459841 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:49.460008 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:49.499073 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.131:34441
I20240507 08:06:49.499166 32416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.131:34441 every 8 connection(s)
I20240507 08:06:49.503834 28050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:06:49.512732 32420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:06:49.513662 32421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:49.515077 32417 heartbeater.cc:347] Connected to a master server at 127.27.100.190:42731
W20240507 08:06:49.515389 32423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:06:49.515523 32417 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:49.516464 32417 heartbeater.cc:511] Master 127.27.100.190:42731 requested a full tablet report, sending...
I20240507 08:06:49.516817 28050 server_base.cc:1034] running on GCE node
I20240507 08:06:49.517663 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:06:49.517886 28050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:06:49.518056 28050 hybrid_clock.cc:648] HybridClock initialized: now 1715069209518038 us; error 0 us; skew 500 ppm
I20240507 08:06:49.518678 32141 ts_manager.cc:194] Registered new tserver with Master: b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131:34441)
I20240507 08:06:49.518823 28050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:06:49.520615 32141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37734
I20240507 08:06:49.521832 28050 webserver.cc:460] Webserver started at http://127.27.100.132:33227/ using document root <none> and password file <none>
I20240507 08:06:49.522444 28050 fs_manager.cc:359] Metadata directory not provided
I20240507 08:06:49.522712 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:06:49.523026 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:06:49.524108 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-3-root/instance:
uuid: "51410f52dd134b8e85e1cc843a1bd34c"
format_stamp: "Formatted at 2024-05-07 08:06:49 on dist-test-slave-j3cx"
I20240507 08:06:49.528460 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.004s
I20240507 08:06:49.531955 32428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:49.532913 28050 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240507 08:06:49.533288 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-3-root
uuid: "51410f52dd134b8e85e1cc843a1bd34c"
format_stamp: "Formatted at 2024-05-07 08:06:49 on dist-test-slave-j3cx"
I20240507 08:06:49.533571 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:06:49.547154 28050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240507 08:06:49.548290 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:06:49.549706 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:06:49.551921 28050 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:06:49.552120 28050 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:49.552343 28050 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:06:49.552500 28050 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240507 08:06:49.590564 28050 rpc_server.cc:307] RPC server started. Bound to: 127.27.100.132:38847
I20240507 08:06:49.590660 32490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.100.132:38847 every 8 connection(s)
I20240507 08:06:49.603736 32491 heartbeater.cc:347] Connected to a master server at 127.27.100.190:42731
I20240507 08:06:49.604127 32491 heartbeater.cc:464] Registering TS with master...
I20240507 08:06:49.604868 32491 heartbeater.cc:511] Master 127.27.100.190:42731 requested a full tablet report, sending...
I20240507 08:06:49.606575 32141 ts_manager.cc:194] Registered new tserver with Master: 51410f52dd134b8e85e1cc843a1bd34c (127.27.100.132:38847)
I20240507 08:06:49.607218 28050 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.013693699s
I20240507 08:06:49.608299 32141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37748
I20240507 08:06:49.627123 28050 test_util.cc:268] Using random seed: -1425978915
I20240507 08:06:49.647958 32141 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37758:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20240507 08:06:49.694129 32308 tablet_service.cc:1467] Processing CreateTablet for tablet f073d618b44c40ffad4d8c5ebdde3352 (DEFAULT_TABLE table=test-workload [id=3f7fcac77b194b88af2e7d544690ed79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:49.695477 32308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f073d618b44c40ffad4d8c5ebdde3352. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:49.700495 32382 tablet_service.cc:1467] Processing CreateTablet for tablet f073d618b44c40ffad4d8c5ebdde3352 (DEFAULT_TABLE table=test-workload [id=3f7fcac77b194b88af2e7d544690ed79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:49.701850 32382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f073d618b44c40ffad4d8c5ebdde3352. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:49.705286 32233 tablet_service.cc:1467] Processing CreateTablet for tablet f073d618b44c40ffad4d8c5ebdde3352 (DEFAULT_TABLE table=test-workload [id=3f7fcac77b194b88af2e7d544690ed79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:06:49.706576 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f073d618b44c40ffad4d8c5ebdde3352. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:49.711891 32511 tablet_bootstrap.cc:492] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8: Bootstrap starting.
I20240507 08:06:49.716421 32512 tablet_bootstrap.cc:492] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6: Bootstrap starting.
I20240507 08:06:49.718907 32511 tablet_bootstrap.cc:654] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:49.723845 32512 tablet_bootstrap.cc:654] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:49.734259 32511 tablet_bootstrap.cc:492] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8: No bootstrap required, opened a new log
I20240507 08:06:49.734788 32511 ts_tablet_manager.cc:1367] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8: Time spent bootstrapping tablet: real 0.023s	user 0.006s	sys 0.007s
I20240507 08:06:49.735751 32512 tablet_bootstrap.cc:492] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6: No bootstrap required, opened a new log
I20240507 08:06:49.735780 32514 tablet_bootstrap.cc:492] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c: Bootstrap starting.
I20240507 08:06:49.736275 32512 ts_tablet_manager.cc:1367] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6: Time spent bootstrapping tablet: real 0.020s	user 0.004s	sys 0.015s
I20240507 08:06:49.737603 32511 raft_consensus.cc:348] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } }
I20240507 08:06:49.738446 32511 raft_consensus.cc:374] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:49.738263 32512 raft_consensus.cc:348] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } }
I20240507 08:06:49.738830 32511 raft_consensus.cc:729] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ec0da746b2424dbd6c481df2d336b8, State: Initialized, Role: FOLLOWER
I20240507 08:06:49.738991 32512 raft_consensus.cc:374] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:49.739403 32512 raft_consensus.cc:729] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b434dfb0864644ecbcc7cb4a8e68e3a6, State: Initialized, Role: FOLLOWER
I20240507 08:06:49.739789 32511 consensus_queue.cc:260] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } }
I20240507 08:06:49.740094 32512 consensus_queue.cc:260] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } }
I20240507 08:06:49.742642 32514 tablet_bootstrap.cc:654] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c: Neither blocks nor log segments found. Creating new log.
I20240507 08:06:49.745215 32511 ts_tablet_manager.cc:1398] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8: Time spent starting tablet: real 0.010s	user 0.000s	sys 0.011s
I20240507 08:06:49.745342 32343 heartbeater.cc:503] Master 127.27.100.190:42731 was elected leader, sending a full tablet report...
I20240507 08:06:49.746109 32512 ts_tablet_manager.cc:1398] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.002s
I20240507 08:06:49.746780 32417 heartbeater.cc:503] Master 127.27.100.190:42731 was elected leader, sending a full tablet report...
I20240507 08:06:49.750371 32514 tablet_bootstrap.cc:492] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c: No bootstrap required, opened a new log
I20240507 08:06:49.750878 32514 ts_tablet_manager.cc:1367] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.006s
I20240507 08:06:49.753544 32514 raft_consensus.cc:348] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } }
I20240507 08:06:49.754266 32514 raft_consensus.cc:374] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:06:49.754542 32514 raft_consensus.cc:729] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb033d0ab28c460e94b755526f99df7c, State: Initialized, Role: FOLLOWER
I20240507 08:06:49.755228 32514 consensus_queue.cc:260] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } }
I20240507 08:06:49.757810 32268 heartbeater.cc:503] Master 127.27.100.190:42731 was elected leader, sending a full tablet report...
I20240507 08:06:49.758316 32514 ts_tablet_manager.cc:1398] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20240507 08:06:50.047276 32517 raft_consensus.cc:482] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:50.047829 32517 raft_consensus.cc:504] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } }
I20240507 08:06:50.049664 32517 leader_election.cc:288] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20ec0da746b2424dbd6c481df2d336b8 (127.27.100.130:44247), cb033d0ab28c460e94b755526f99df7c (127.27.100.129:36785)
I20240507 08:06:50.057889 32516 raft_consensus.cc:482] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:50.058432 32516 raft_consensus.cc:504] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } }
I20240507 08:06:50.058861 32318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f073d618b44c40ffad4d8c5ebdde3352" candidate_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ec0da746b2424dbd6c481df2d336b8" is_pre_election: true
I20240507 08:06:50.059484 32243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f073d618b44c40ffad4d8c5ebdde3352" candidate_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb033d0ab28c460e94b755526f99df7c" is_pre_election: true
I20240507 08:06:50.059815 32318 raft_consensus.cc:2454] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b434dfb0864644ecbcc7cb4a8e68e3a6 in term 0.
I20240507 08:06:50.060210 32243 raft_consensus.cc:2454] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b434dfb0864644ecbcc7cb4a8e68e3a6 in term 0.
I20240507 08:06:50.061162 32355 leader_election.cc:302] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ec0da746b2424dbd6c481df2d336b8, b434dfb0864644ecbcc7cb4a8e68e3a6; no voters: 
I20240507 08:06:50.062072 32517 raft_consensus.cc:2781] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:06:50.062542 32517 raft_consensus.cc:482] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:50.062909 32517 raft_consensus.cc:3037] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:50.064311 32516 leader_election.cc:288] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb033d0ab28c460e94b755526f99df7c (127.27.100.129:36785), b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131:34441)
I20240507 08:06:50.069474 32517 raft_consensus.cc:504] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } }
I20240507 08:06:50.072212 32318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f073d618b44c40ffad4d8c5ebdde3352" candidate_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ec0da746b2424dbd6c481df2d336b8"
I20240507 08:06:50.073042 32318 raft_consensus.cc:3037] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:50.072213 32243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f073d618b44c40ffad4d8c5ebdde3352" candidate_uuid: "20ec0da746b2424dbd6c481df2d336b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb033d0ab28c460e94b755526f99df7c" is_pre_election: true
I20240507 08:06:50.073657 32392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f073d618b44c40ffad4d8c5ebdde3352" candidate_uuid: "20ec0da746b2424dbd6c481df2d336b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" is_pre_election: true
I20240507 08:06:50.074236 32243 raft_consensus.cc:2454] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20ec0da746b2424dbd6c481df2d336b8 in term 0.
I20240507 08:06:50.074513 32392 raft_consensus.cc:2379] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20ec0da746b2424dbd6c481df2d336b8 in current term 1: Already voted for candidate b434dfb0864644ecbcc7cb4a8e68e3a6 in this term.
I20240507 08:06:50.074995 32242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f073d618b44c40ffad4d8c5ebdde3352" candidate_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb033d0ab28c460e94b755526f99df7c"
I20240507 08:06:50.074535 32517 leader_election.cc:288] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [CANDIDATE]: Term 1 election: Requested vote from peers 20ec0da746b2424dbd6c481df2d336b8 (127.27.100.130:44247), cb033d0ab28c460e94b755526f99df7c (127.27.100.129:36785)
I20240507 08:06:50.075630 32242 raft_consensus.cc:3037] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:06:50.075614 32284 leader_election.cc:302] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ec0da746b2424dbd6c481df2d336b8, cb033d0ab28c460e94b755526f99df7c; no voters: 
I20240507 08:06:50.080214 32318 raft_consensus.cc:2454] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b434dfb0864644ecbcc7cb4a8e68e3a6 in term 1.
I20240507 08:06:50.080744 32516 raft_consensus.cc:2741] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240507 08:06:50.081308 32355 leader_election.cc:302] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ec0da746b2424dbd6c481df2d336b8, b434dfb0864644ecbcc7cb4a8e68e3a6; no voters: 
I20240507 08:06:50.082180 32517 raft_consensus.cc:2781] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:06:50.082930 32242 raft_consensus.cc:2454] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b434dfb0864644ecbcc7cb4a8e68e3a6 in term 1.
I20240507 08:06:50.083367 32517 raft_consensus.cc:686] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [term 1 LEADER]: Becoming Leader. State: Replica: b434dfb0864644ecbcc7cb4a8e68e3a6, State: Running, Role: LEADER
I20240507 08:06:50.084283 32517 consensus_queue.cc:237] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } }
I20240507 08:06:50.091862 32140 catalog_manager.cc:5484] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131). New cstate: current_term: 1 leader_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:50.385953 32308 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:50.396454 32456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:50.397897 32233 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:50.400925 32382 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240507 08:06:50.610780 32491 heartbeater.cc:503] Master 127.27.100.190:42731 was elected leader, sending a full tablet report...
I20240507 08:06:50.618851 32557 consensus_queue.cc:1035] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:06:50.659767 32564 consensus_queue.cc:1035] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 a204481d2a5a4d8e9b0d8e035546ad16 | 127.27.100.190:42731 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 20ec0da746b2424dbd6c481df2d336b8 | 127.27.100.130:44247 | HEALTHY | <none>   |       0        |       0
 51410f52dd134b8e85e1cc843a1bd34c | 127.27.100.132:38847 | HEALTHY | <none>   |       0        |       0
 b434dfb0864644ecbcc7cb4a8e68e3a6 | 127.27.100.131:34441 | HEALTHY | <none>   |       1        |       0
 cb033d0ab28c460e94b755526f99df7c | 127.27.100.129:36785 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
Tablet f073d618b44c40ffad4d8c5ebdde3352 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
  20ec0da746b2424dbd6c481df2d336b8 (127.27.100.130:44247): RUNNING
  cb033d0ab28c460e94b755526f99df7c (127.27.100.129:36785): RUNNING
  b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131:34441): RUNNING [LEADER]
All reported replicas are:
  A = 20ec0da746b2424dbd6c481df2d336b8
  B = cb033d0ab28c460e94b755526f99df7c
  C = b434dfb0864644ecbcc7cb4a8e68e3a6
The consensus matrix is:
 Config source |   Replicas   | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
 master        | A   B   C*   |              |              | Yes
 A             | A   B   C    | 1            | -1           | Yes
 B             | A   B   C    | 1            | -1           | Yes
 C             | A   B   C*   | 1            | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |       Status       | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | CONSENSUS_MISMATCH | 1             | 0       | 0          | 0                | 1

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |         Host         | Replica Count
-------+----------------------------------+----------------------+---------------
 Small | 51410f52dd134b8e85e1cc843a1bd34c | 127.27.100.132:38847 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20240507 08:06:50.674659 28050 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20240507 08:06:51.040755 32456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:51.044499 32308 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:51.050697 32382 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240507 08:06:51.052964 32233 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 a204481d2a5a4d8e9b0d8e035546ad16 | 127.27.100.190:42731 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 20ec0da746b2424dbd6c481df2d336b8 | 127.27.100.130:44247 | HEALTHY | <none>   |       0        |       0
 51410f52dd134b8e85e1cc843a1bd34c | 127.27.100.132:38847 | HEALTHY | <none>   |       0        |       0
 b434dfb0864644ecbcc7cb4a8e68e3a6 | 127.27.100.131:34441 | HEALTHY | <none>   |       1        |       0
 cb033d0ab28c460e94b755526f99df7c | 127.27.100.129:36785 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
     Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |         Host         | Replica Count
-------+----------------------------------+----------------------+---------------
 Small | 51410f52dd134b8e85e1cc843a1bd34c | 127.27.100.132:38847 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240507 08:06:51.282362 28050 log_verifier.cc:126] Checking tablet f073d618b44c40ffad4d8c5ebdde3352
I20240507 08:06:51.341884 28050 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f073d618b44c40ffad4d8c5ebdde3352
I20240507 08:06:51.343786 28050 tablet_replica.cc:330] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6: stopping tablet replica
I20240507 08:06:51.344347 28050 raft_consensus.cc:2229] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:51.345146 28050 raft_consensus.cc:2258] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:51.544113 32308 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:51.551034 32382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:51.553457 32456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:51.554962 32233 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 a204481d2a5a4d8e9b0d8e035546ad16 | 127.27.100.190:42731 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 20ec0da746b2424dbd6c481df2d336b8 | 127.27.100.130:44247 | HEALTHY | <none>   |       0        |       0
 51410f52dd134b8e85e1cc843a1bd34c | 127.27.100.132:38847 | HEALTHY | <none>   |       0        |       0
 b434dfb0864644ecbcc7cb4a8e68e3a6 | 127.27.100.131:34441 | HEALTHY | <none>   |       0        |       0
 cb033d0ab28c460e94b755526f99df7c | 127.27.100.129:36785 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
Tablet f073d618b44c40ffad4d8c5ebdde3352 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  20ec0da746b2424dbd6c481df2d336b8 (127.27.100.130:44247): RUNNING
  cb033d0ab28c460e94b755526f99df7c (127.27.100.129:36785): RUNNING
  b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131:34441): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
  A = 20ec0da746b2424dbd6c481df2d336b8
  B = cb033d0ab28c460e94b755526f99df7c
  C = b434dfb0864644ecbcc7cb4a8e68e3a6
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A   B   C*             |              |              | Yes
 A             | A   B   C*             | 1            | -1           | Yes
 B             | A   B   C*             | 1            | -1           | Yes
 C             | [config not available] |              |              | 

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |         Host         | Replica Count
-------+----------------------------------+----------------------+---------------
 Small | 51410f52dd134b8e85e1cc843a1bd34c | 127.27.100.132:38847 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20240507 08:06:51.779811 28050 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20240507 08:06:52.132961 32308 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:52.136873 32456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:52.140779 32382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:52.148229 32233 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 a204481d2a5a4d8e9b0d8e035546ad16 | 127.27.100.190:42731 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 20ec0da746b2424dbd6c481df2d336b8 | 127.27.100.130:44247 | HEALTHY | <none>   |       0        |       0
 51410f52dd134b8e85e1cc843a1bd34c | 127.27.100.132:38847 | HEALTHY | <none>   |       0        |       0
 b434dfb0864644ecbcc7cb4a8e68e3a6 | 127.27.100.131:34441 | HEALTHY | <none>   |       0        |       0
 cb033d0ab28c460e94b755526f99df7c | 127.27.100.129:36785 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
Tablet f073d618b44c40ffad4d8c5ebdde3352 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  20ec0da746b2424dbd6c481df2d336b8 (127.27.100.130:44247): RUNNING
  cb033d0ab28c460e94b755526f99df7c (127.27.100.129:36785): RUNNING
  b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131:34441): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
  A = 20ec0da746b2424dbd6c481df2d336b8
  B = cb033d0ab28c460e94b755526f99df7c
  C = b434dfb0864644ecbcc7cb4a8e68e3a6
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A   B   C*             |              |              | Yes
 A             | A   B   C*             | 1            | -1           | Yes
 B             | A   B   C*             | 1            | -1           | Yes
 C             | [config not available] |              |              | 

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |         Host         | Replica Count
-------+----------------------------------+----------------------+---------------
 Small | 51410f52dd134b8e85e1cc843a1bd34c | 127.27.100.132:38847 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20240507 08:06:52.361086 28050 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20240507 08:06:52.556471 32650 raft_consensus.cc:482] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b434dfb0864644ecbcc7cb4a8e68e3a6)
I20240507 08:06:52.556913 32650 raft_consensus.cc:504] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } }
I20240507 08:06:52.558354 32650 leader_election.cc:288] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cb033d0ab28c460e94b755526f99df7c (127.27.100.129:36785), b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131:34441)
I20240507 08:06:52.559351 32392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f073d618b44c40ffad4d8c5ebdde3352" candidate_uuid: "20ec0da746b2424dbd6c481df2d336b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" is_pre_election: true
I20240507 08:06:52.559329 32242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f073d618b44c40ffad4d8c5ebdde3352" candidate_uuid: "20ec0da746b2424dbd6c481df2d336b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb033d0ab28c460e94b755526f99df7c" is_pre_election: true
W20240507 08:06:52.560858 32283 leader_election.cc:341] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131:34441): Illegal state: cannot vote while shut down
I20240507 08:06:52.561327 32284 leader_election.cc:302] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20ec0da746b2424dbd6c481df2d336b8; no voters: b434dfb0864644ecbcc7cb4a8e68e3a6, cb033d0ab28c460e94b755526f99df7c
I20240507 08:06:52.561911 32650 raft_consensus.cc:2726] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:06:52.737208 32668 raft_consensus.cc:482] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b434dfb0864644ecbcc7cb4a8e68e3a6)
I20240507 08:06:52.737824 32668 raft_consensus.cc:504] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } }
I20240507 08:06:52.740188 32668 leader_election.cc:288] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20ec0da746b2424dbd6c481df2d336b8 (127.27.100.130:44247), b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131:34441)
I20240507 08:06:52.767700 32318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f073d618b44c40ffad4d8c5ebdde3352" candidate_uuid: "cb033d0ab28c460e94b755526f99df7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "20ec0da746b2424dbd6c481df2d336b8" is_pre_election: true
I20240507 08:06:52.768286 32318 raft_consensus.cc:2454] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb033d0ab28c460e94b755526f99df7c in term 1.
I20240507 08:06:52.769173 32206 leader_election.cc:302] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ec0da746b2424dbd6c481df2d336b8, cb033d0ab28c460e94b755526f99df7c; no voters: 
I20240507 08:06:52.769861 32668 raft_consensus.cc:2781] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:06:52.770246 32668 raft_consensus.cc:482] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 1 FOLLOWER]: Starting leader election (detected failure of leader b434dfb0864644ecbcc7cb4a8e68e3a6)
I20240507 08:06:52.770594 32668 raft_consensus.cc:3037] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:06:52.771637 32392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f073d618b44c40ffad4d8c5ebdde3352" candidate_uuid: "cb033d0ab28c460e94b755526f99df7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" is_pre_election: true
W20240507 08:06:52.774564 32208 leader_election.cc:341] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131:34441): Illegal state: cannot vote while shut down
I20240507 08:06:52.777245 32668 raft_consensus.cc:504] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } }
I20240507 08:06:52.778990 32668 leader_election.cc:288] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [CANDIDATE]: Term 2 election: Requested vote from peers 20ec0da746b2424dbd6c481df2d336b8 (127.27.100.130:44247), b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131:34441)
I20240507 08:06:52.779723 32318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f073d618b44c40ffad4d8c5ebdde3352" candidate_uuid: "cb033d0ab28c460e94b755526f99df7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "20ec0da746b2424dbd6c481df2d336b8"
I20240507 08:06:52.779955 32392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f073d618b44c40ffad4d8c5ebdde3352" candidate_uuid: "cb033d0ab28c460e94b755526f99df7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6"
I20240507 08:06:52.780303 32318 raft_consensus.cc:3037] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:06:52.781102 32208 leader_election.cc:341] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131:34441): Illegal state: cannot vote while shut down
I20240507 08:06:52.785446 32318 raft_consensus.cc:2454] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb033d0ab28c460e94b755526f99df7c in term 2.
I20240507 08:06:52.786432 32206 leader_election.cc:302] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20ec0da746b2424dbd6c481df2d336b8, cb033d0ab28c460e94b755526f99df7c; no voters: b434dfb0864644ecbcc7cb4a8e68e3a6
I20240507 08:06:52.787221 32668 raft_consensus.cc:2781] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:06:52.789822 32668 raft_consensus.cc:686] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 2 LEADER]: Becoming Leader. State: Replica: cb033d0ab28c460e94b755526f99df7c, State: Running, Role: LEADER
I20240507 08:06:52.790868 32668 consensus_queue.cc:237] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } }
I20240507 08:06:52.818848 32456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:52.822575 32140 catalog_manager.cc:5484] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c reported cstate change: term changed from 1 to 2, leader changed from b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131) to cb033d0ab28c460e94b755526f99df7c (127.27.100.129). New cstate: current_term: 2 leader_uuid: "cb033d0ab28c460e94b755526f99df7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b434dfb0864644ecbcc7cb4a8e68e3a6" member_type: VOTER last_known_addr { host: "127.27.100.131" port: 34441 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:52.837205 32382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:52.838629 32308 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:52.853027 32233 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 a204481d2a5a4d8e9b0d8e035546ad16 | 127.27.100.190:42731 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 20ec0da746b2424dbd6c481df2d336b8 | 127.27.100.130:44247 | HEALTHY | <none>   |       0        |       0
 51410f52dd134b8e85e1cc843a1bd34c | 127.27.100.132:38847 | HEALTHY | <none>   |       0        |       0
 b434dfb0864644ecbcc7cb4a8e68e3a6 | 127.27.100.131:34441 | HEALTHY | <none>   |       0        |       0
 cb033d0ab28c460e94b755526f99df7c | 127.27.100.129:36785 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
Tablet f073d618b44c40ffad4d8c5ebdde3352 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  20ec0da746b2424dbd6c481df2d336b8 (127.27.100.130:44247): RUNNING
  cb033d0ab28c460e94b755526f99df7c (127.27.100.129:36785): RUNNING
  b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131:34441): not running [LEADER]
    State:       FAILED
    Data state:  TABLET_DATA_READY
    Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
  A = 20ec0da746b2424dbd6c481df2d336b8
  B = cb033d0ab28c460e94b755526f99df7c
  C = b434dfb0864644ecbcc7cb4a8e68e3a6
The consensus matrix is:
 Config source |        Replicas        | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
 master        | A   B   C*             |              |              | Yes
 A             | A   B   C              | 2            | -1           | Yes
 B             | A   B*  C              | 2            | -1           | Yes
 C             | [config not available] |              |              | 

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |         Host         | Replica Count
-------+----------------------------------+----------------------+---------------
 Small | 51410f52dd134b8e85e1cc843a1bd34c | 127.27.100.132:38847 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20240507 08:06:53.066711 28050 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20240507 08:06:53.304833 32208 consensus_queue.cc:579] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [LEADER]: The tablet replica hosted on peer b434dfb0864644ecbcc7cb4a8e68e3a6 has failed
W20240507 08:06:53.305305 32208 consensus_peers.cc:485] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c -> Peer b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131:34441): Couldn't send request to peer b434dfb0864644ecbcc7cb4a8e68e3a6. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:53.348196 32318 raft_consensus.cc:1261] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 2 FOLLOWER]: Refusing update from remote peer cb033d0ab28c460e94b755526f99df7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:06:53.349370 32668 consensus_queue.cc:1035] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:06:53.366092 32242 consensus_queue.cc:237] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } }
I20240507 08:06:53.369570 32318 raft_consensus.cc:1261] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 2 FOLLOWER]: Refusing update from remote peer cb033d0ab28c460e94b755526f99df7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240507 08:06:53.370626 32693 consensus_queue.cc:1035] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:53.375353 32688 raft_consensus.cc:2932] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } }
I20240507 08:06:53.376657 32318 raft_consensus.cc:2932] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } }
I20240507 08:06:53.381695 32128 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f073d618b44c40ffad4d8c5ebdde3352 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:06:53.384613 32139 catalog_manager.cc:5484] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c reported cstate change: config changed from index -1 to 3, VOTER b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131) evicted. New cstate: current_term: 2 leader_uuid: "cb033d0ab28c460e94b755526f99df7c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:53.395062 32382 tablet_service.cc:1514] Processing DeleteTablet for tablet f073d618b44c40ffad4d8c5ebdde3352 with delete_type TABLET_DATA_TOMBSTONED (TS b434dfb0864644ecbcc7cb4a8e68e3a6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40506
I20240507 08:06:53.397567 32697 ts_tablet_manager.cc:1875] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:06:53.398358 32242 consensus_queue.cc:237] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "51410f52dd134b8e85e1cc843a1bd34c" member_type: NON_VOTER last_known_addr { host: "127.27.100.132" port: 38847 } attrs { promote: true } }
I20240507 08:06:53.404482 32318 raft_consensus.cc:1261] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 2 FOLLOWER]: Refusing update from remote peer cb033d0ab28c460e94b755526f99df7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240507 08:06:53.406023 32668 consensus_queue.cc:1035] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:53.414063 32668 raft_consensus.cc:2932] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER 51410f52dd134b8e85e1cc843a1bd34c (127.27.100.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "51410f52dd134b8e85e1cc843a1bd34c" member_type: NON_VOTER last_known_addr { host: "127.27.100.132" port: 38847 } attrs { promote: true } } }
I20240507 08:06:53.417019 32697 ts_tablet_manager.cc:1888] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240507 08:06:53.417553 32697 log.cc:1196] T f073d618b44c40ffad4d8c5ebdde3352 P b434dfb0864644ecbcc7cb4a8e68e3a6: Deleting WAL directory at /tmp/dist-test-taskpJJpQT/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1715069146555183-28050-0/minicluster-data/ts-2-root/wals/f073d618b44c40ffad4d8c5ebdde3352
I20240507 08:06:53.417330 32317 raft_consensus.cc:2932] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER 51410f52dd134b8e85e1cc843a1bd34c (127.27.100.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "51410f52dd134b8e85e1cc843a1bd34c" member_type: NON_VOTER last_known_addr { host: "127.27.100.132" port: 38847 } attrs { promote: true } } }
I20240507 08:06:53.419843 32127 catalog_manager.cc:4830] TS b434dfb0864644ecbcc7cb4a8e68e3a6 (127.27.100.131:34441): tablet f073d618b44c40ffad4d8c5ebdde3352 (table test-workload [id=3f7fcac77b194b88af2e7d544690ed79]) successfully deleted
I20240507 08:06:53.424921 32128 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f073d618b44c40ffad4d8c5ebdde3352 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240507 08:06:53.428823 32207 consensus_peers.cc:485] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c -> Peer 51410f52dd134b8e85e1cc843a1bd34c (127.27.100.132:38847): Couldn't send request to peer 51410f52dd134b8e85e1cc843a1bd34c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f073d618b44c40ffad4d8c5ebdde3352. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:53.428189 32139 catalog_manager.cc:5484] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c reported cstate change: config changed from index 3 to 4, NON_VOTER 51410f52dd134b8e85e1cc843a1bd34c (127.27.100.132) added. New cstate: current_term: 2 leader_uuid: "cb033d0ab28c460e94b755526f99df7c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51410f52dd134b8e85e1cc843a1bd34c" member_type: NON_VOTER last_known_addr { host: "127.27.100.132" port: 38847 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:53.640697 32308 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:53.641623 32456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:53.648061 32382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240507 08:06:53.649693 32233 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240507 08:06:53.839435 32729 ts_tablet_manager.cc:902] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c: Initiating tablet copy from peer cb033d0ab28c460e94b755526f99df7c (127.27.100.129:36785)
I20240507 08:06:53.841092 32729 tablet_copy_client.cc:316] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.100.129:36785
I20240507 08:06:53.850251 32253 tablet_copy_service.cc:140] P cb033d0ab28c460e94b755526f99df7c: Received BeginTabletCopySession request for tablet f073d618b44c40ffad4d8c5ebdde3352 from peer 51410f52dd134b8e85e1cc843a1bd34c ({username='slave'} at 127.0.0.1:39824)
I20240507 08:06:53.850721 32253 tablet_copy_service.cc:161] P cb033d0ab28c460e94b755526f99df7c: Beginning new tablet copy session on tablet f073d618b44c40ffad4d8c5ebdde3352 from peer 51410f52dd134b8e85e1cc843a1bd34c at {username='slave'} at 127.0.0.1:39824: session id = 51410f52dd134b8e85e1cc843a1bd34c-f073d618b44c40ffad4d8c5ebdde3352
I20240507 08:06:53.856029 32253 tablet_copy_source_session.cc:205] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:06:53.858574 32729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f073d618b44c40ffad4d8c5ebdde3352. 1 dirs total, 0 dirs full, 0 dirs failed
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 a204481d2a5a4d8e9b0d8e035546ad16 | 127.27.100.190:42731 | HEALTHY

Unusual flags for Master:
               Flag               | Value |  Tags  |         Master
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
 time_source         | system_unsync                                               | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 20ec0da746b2424dbd6c481df2d336b8 | 127.27.100.130:44247 | HEALTHY | <none>   |       0        |       0
 51410f52dd134b8e85e1cc843a1bd34c | 127.27.100.132:38847 | HEALTHY | <none>   |       0        |       0
 b434dfb0864644ecbcc7cb4a8e68e3a6 | 127.27.100.131:34441 | HEALTHY | <none>   |       0        |       0
 cb033d0ab28c460e94b755526f99df7c | 127.27.100.129:36785 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
               Flag               | Value |  Tags  |      Tablet Server
----------------------------------+-------+--------+-------------------------
 rpc_server_allow_ephemeral_ports | true  | unsafe | all 4 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
 time_source         | system_unsync                                               | all 4 server(s) checked

Version Summary
I20240507 08:06:53.870425 32729 tablet_copy_client.cc:791] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c: tablet copy: Starting download of 0 data blocks...
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
I20240507 08:06:53.871023 32729 tablet_copy_client.cc:655] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c: tablet copy: Starting download of 1 WAL segments...
     Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
 test-workload | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Tablet Replica Count Outliers
 Type  |               UUID               |         Host         | Replica Count
-------+----------------------------------+----------------------+---------------
 Small | b434dfb0864644ecbcc7cb4a8e68e3a6 | 127.27.100.131:34441 | 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240507 08:06:53.875447 32729 tablet_copy_client.cc:531] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:06:53.878881 28050 log_verifier.cc:126] Checking tablet f073d618b44c40ffad4d8c5ebdde3352
I20240507 08:06:53.883405 32729 tablet_bootstrap.cc:492] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c: Bootstrap starting.
I20240507 08:06:53.902825 32729 tablet_bootstrap.cc:492] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:06:53.903473 32729 tablet_bootstrap.cc:492] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c: Bootstrap complete.
I20240507 08:06:53.904131 32729 ts_tablet_manager.cc:1367] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c: Time spent bootstrapping tablet: real 0.021s	user 0.008s	sys 0.015s
I20240507 08:06:53.907090 32729 raft_consensus.cc:348] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "51410f52dd134b8e85e1cc843a1bd34c" member_type: NON_VOTER last_known_addr { host: "127.27.100.132" port: 38847 } attrs { promote: true } }
I20240507 08:06:53.907878 32729 raft_consensus.cc:729] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 51410f52dd134b8e85e1cc843a1bd34c, State: Initialized, Role: LEARNER
I20240507 08:06:53.908576 32729 consensus_queue.cc:260] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20ec0da746b2424dbd6c481df2d336b8" member_type: VOTER last_known_addr { host: "127.27.100.130" port: 44247 } } peers { permanent_uuid: "cb033d0ab28c460e94b755526f99df7c" member_type: VOTER last_known_addr { host: "127.27.100.129" port: 36785 } } peers { permanent_uuid: "51410f52dd134b8e85e1cc843a1bd34c" member_type: NON_VOTER last_known_addr { host: "127.27.100.132" port: 38847 } attrs { promote: true } }
I20240507 08:06:53.911257 32729 ts_tablet_manager.cc:1398] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.002s
I20240507 08:06:53.913372 32253 tablet_copy_service.cc:342] P cb033d0ab28c460e94b755526f99df7c: Request end of tablet copy session 51410f52dd134b8e85e1cc843a1bd34c-f073d618b44c40ffad4d8c5ebdde3352 received from {username='slave'} at 127.0.0.1:39824
I20240507 08:06:53.913885 32253 tablet_copy_service.cc:433] P cb033d0ab28c460e94b755526f99df7c: ending tablet copy session 51410f52dd134b8e85e1cc843a1bd34c-f073d618b44c40ffad4d8c5ebdde3352 on tablet f073d618b44c40ffad4d8c5ebdde3352 with peer 51410f52dd134b8e85e1cc843a1bd34c
I20240507 08:06:53.949522 28050 log_verifier.cc:177] Verified matching terms for 4 ops in tablet f073d618b44c40ffad4d8c5ebdde3352
I20240507 08:06:53.951795 28050 tablet_server.cc:178] TabletServer@127.27.100.129:0 shutting down...
I20240507 08:06:53.971237 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:53.971932 28050 tablet_replica.cc:330] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c: stopping tablet replica
I20240507 08:06:53.972409 28050 raft_consensus.cc:2229] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 2 LEADER]: Raft consensus shutting down.
I20240507 08:06:53.973210 28050 raft_consensus.cc:2258] T f073d618b44c40ffad4d8c5ebdde3352 P cb033d0ab28c460e94b755526f99df7c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:53.992913 28050 tablet_server.cc:194] TabletServer@127.27.100.129:0 shutdown complete.
I20240507 08:06:54.001693 28050 tablet_server.cc:178] TabletServer@127.27.100.130:0 shutting down...
I20240507 08:06:54.019392 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:54.020009 28050 tablet_replica.cc:330] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8: stopping tablet replica
I20240507 08:06:54.020493 28050 raft_consensus.cc:2229] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:54.021005 28050 raft_consensus.cc:2258] T f073d618b44c40ffad4d8c5ebdde3352 P 20ec0da746b2424dbd6c481df2d336b8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:54.040673 28050 tablet_server.cc:194] TabletServer@127.27.100.130:0 shutdown complete.
I20240507 08:06:54.049149 28050 tablet_server.cc:178] TabletServer@127.27.100.131:0 shutting down...
I20240507 08:06:54.066138 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:54.083026 28050 tablet_server.cc:194] TabletServer@127.27.100.131:0 shutdown complete.
I20240507 08:06:54.091609 28050 tablet_server.cc:178] TabletServer@127.27.100.132:0 shutting down...
I20240507 08:06:54.110092 28050 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:06:54.110858 28050 tablet_replica.cc:330] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c: stopping tablet replica
I20240507 08:06:54.111294 28050 raft_consensus.cc:2229] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c [term 2 LEARNER]: Raft consensus shutting down.
I20240507 08:06:54.111660 28050 raft_consensus.cc:2258] T f073d618b44c40ffad4d8c5ebdde3352 P 51410f52dd134b8e85e1cc843a1bd34c [term 2 LEARNER]: Raft consensus is shut down!
I20240507 08:06:54.130357 28050 tablet_server.cc:194] TabletServer@127.27.100.132:0 shutdown complete.
I20240507 08:06:54.139099 28050 master.cc:537] Master@127.27.100.190:42731 shutting down...
I20240507 08:06:54.153165 28050 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:06:54.153625 28050 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:54.153870 28050 tablet_replica.cc:330] T 00000000000000000000000000000000 P a204481d2a5a4d8e9b0d8e035546ad16: stopping tablet replica
I20240507 08:06:54.170555 28050 master.cc:552] Master@127.27.100.190:42731 shutdown complete.
[       OK ] FailedTabletsAreReplacedITest.OneReplica/1 (5207 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (9652 ms total)

[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (67484 ms total)
[  PASSED  ] 11 tests.
[  SKIPPED ] 1 test, listed below:
[  SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
[  FAILED  ] 1 test, listed below:
[  FAILED  ] TsTabletManagerITest.TestTableStats

 1 FAILED TEST
I20240507 08:06:54.205626 28050 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~10 seconds ago
I20240507 08:06:54.205965 28050 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:644: suppressed but not reported on 84 messages since previous log ~59 seconds ago
I20240507 08:06:54.206143 28050 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 5 messages since previous log ~14 seconds ago
I20240507 08:06:54.206418 28050 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2369: suppressed but not reported on 8 messages since previous log ~13 seconds ago