[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 22:53:35.032409 5542 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 1.210s user 0.390s sys 0.818s
W20260809 22:53:35.033067 5542 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 1.211s user 0.390s sys 0.818s
I20260809 22:53:35.048163 5542 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.5.105.190:38499
I20260809 22:53:35.050793 5542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:53:35.052459 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:35.070571 5555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:35.072405 5554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:35.232141 5557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:35.232414 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:35.233839 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:35.234059 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:35.234300 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316015234260 us; error 0 us; skew 500 ppm
I20260809 22:53:35.235198 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:35.242107 5542 webserver.cc:460] Webserver started at http://127.5.105.190:36575/ using document root <none> and password file <none>
I20260809 22:53:35.243220 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:35.243456 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:35.243939 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:35.252472 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/master-0-root/instance:
uuid: "c87e01b30e244dff8486756002730b00"
format_stamp: "Formatted at 2026-08-09 22:53:35 on dist-test-slave-gp6n"
I20260809 22:53:35.260265 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20260809 22:53:35.265128 5563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:35.266494 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 22:53:35.266837 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/master-0-root
uuid: "c87e01b30e244dff8486756002730b00"
format_stamp: "Formatted at 2026-08-09 22:53:35 on dist-test-slave-gp6n"
I20260809 22:53:35.267184 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:35.295501 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:35.297080 5542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:53:35.297556 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:35.376258 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.190:38499
I20260809 22:53:35.376350 5618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.190:38499 every 8 connection(s)
I20260809 22:53:35.382589 5619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:35.406095 5619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00: Bootstrap starting.
I20260809 22:53:35.412320 5619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:35.414151 5619 log.cc:824] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00: Log is configured to *not* fsync() on all Append() calls
I20260809 22:53:35.418843 5619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00: No bootstrap required, opened a new log
I20260809 22:53:35.441444 5619 raft_consensus.cc:348] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87e01b30e244dff8486756002730b00" member_type: VOTER }
I20260809 22:53:35.442169 5619 raft_consensus.cc:374] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:35.442487 5619 raft_consensus.cc:729] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87e01b30e244dff8486756002730b00, State: Initialized, Role: FOLLOWER
I20260809 22:53:35.443712 5619 consensus_queue.cc:260] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87e01b30e244dff8486756002730b00" member_type: VOTER }
I20260809 22:53:35.444284 5619 raft_consensus.cc:388] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:53:35.444604 5619 raft_consensus.cc:482] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:53:35.444974 5619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:35.449350 5619 raft_consensus.cc:504] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87e01b30e244dff8486756002730b00" member_type: VOTER }
I20260809 22:53:35.450235 5619 leader_election.cc:302] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c87e01b30e244dff8486756002730b00; no voters:
I20260809 22:53:35.452172 5619 leader_election.cc:288] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:53:35.452636 5623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:53:35.455410 5623 raft_consensus.cc:686] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [term 1 LEADER]: Becoming Leader. State: Replica: c87e01b30e244dff8486756002730b00, State: Running, Role: LEADER
I20260809 22:53:35.456238 5623 consensus_queue.cc:237] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87e01b30e244dff8486756002730b00" member_type: VOTER }
I20260809 22:53:35.457172 5619 sys_catalog.cc:564] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:53:35.466312 5624 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c87e01b30e244dff8486756002730b00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87e01b30e244dff8486756002730b00" member_type: VOTER } }
I20260809 22:53:35.466585 5625 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c87e01b30e244dff8486756002730b00. Latest consensus state: current_term: 1 leader_uuid: "c87e01b30e244dff8486756002730b00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87e01b30e244dff8486756002730b00" member_type: VOTER } }
I20260809 22:53:35.467062 5624 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:35.467377 5625 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:35.471060 5633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:53:35.483479 5633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:53:35.487607 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 22:53:35.501813 5633 catalog_manager.cc:1347] Generated new cluster ID: 09fa85be1b764762ac0f2b0b39eda2c5
I20260809 22:53:35.502104 5633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:53:35.542454 5633 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:53:35.544531 5633 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:53:35.563226 5633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00: Generated new TSK 0
I20260809 22:53:35.564126 5633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:53:35.619345 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:35.629146 5645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:35.630964 5647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:35.634671 5542 server_base.cc:1034] running on GCE node
W20260809 22:53:35.634912 5649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:35.635949 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:35.636224 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:35.636417 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316015636402 us; error 0 us; skew 500 ppm
I20260809 22:53:35.637195 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:35.641302 5542 webserver.cc:460] Webserver started at http://127.5.105.129:36463/ using document root <none> and password file <none>
I20260809 22:53:35.641978 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:35.642226 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:35.642580 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:35.644274 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/ts-0-root/instance:
uuid: "797286e6cdb44f17a901a8d4babbb9c0"
format_stamp: "Formatted at 2026-08-09 22:53:35 on dist-test-slave-gp6n"
I20260809 22:53:35.651350 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260809 22:53:35.656502 5654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:35.657331 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260809 22:53:35.657665 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "797286e6cdb44f17a901a8d4babbb9c0"
format_stamp: "Formatted at 2026-08-09 22:53:35 on dist-test-slave-gp6n"
I20260809 22:53:35.658053 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:35.675380 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:35.676681 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:35.679162 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:53:35.683212 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:53:35.683460 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:35.683722 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:53:35.683899 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:35.778434 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:37051
I20260809 22:53:35.779012 5725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:37051 every 8 connection(s)
I20260809 22:53:35.786458 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:35.805343 5730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:35.807485 5731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:35.819583 5542 server_base.cc:1034] running on GCE node
W20260809 22:53:35.820935 5734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:35.822139 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:35.822479 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:35.822706 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316015822692 us; error 0 us; skew 500 ppm
I20260809 22:53:35.823418 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:35.826561 5726 heartbeater.cc:347] Connected to a master server at 127.5.105.190:38499
I20260809 22:53:35.826988 5726 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:35.827486 5542 webserver.cc:460] Webserver started at http://127.5.105.130:45283/ using document root <none> and password file <none>
I20260809 22:53:35.828012 5726 heartbeater.cc:511] Master 127.5.105.190:38499 requested a full tablet report, sending...
I20260809 22:53:35.828096 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:35.828397 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:35.828706 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:35.830118 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/ts-1-root/instance:
uuid: "f9d60226269f41849f171ea50026baa1"
format_stamp: "Formatted at 2026-08-09 22:53:35 on dist-test-slave-gp6n"
I20260809 22:53:35.831570 5581 ts_manager.cc:194] Registered new tserver with Master: 797286e6cdb44f17a901a8d4babbb9c0 (127.5.105.129:37051)
I20260809 22:53:35.833793 5581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60984
I20260809 22:53:35.837494 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260809 22:53:35.843132 5739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:35.843995 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:53:35.844311 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/ts-1-root
uuid: "f9d60226269f41849f171ea50026baa1"
format_stamp: "Formatted at 2026-08-09 22:53:35 on dist-test-slave-gp6n"
I20260809 22:53:35.844667 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786316013627029-5542-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:35.865702 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:35.867340 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:35.870008 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:53:35.874233 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:53:35.874495 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 22:53:35.874771 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:53:35.874991 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:35.937974 5726 heartbeater.cc:503] Master 127.5.105.190:38499 was elected leader, sending a full tablet report...
I20260809 22:53:35.967427 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.130:45365
I20260809 22:53:35.967535 5811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.130:45365 every 8 connection(s)
I20260809 22:53:35.986928 5812 heartbeater.cc:347] Connected to a master server at 127.5.105.190:38499
I20260809 22:53:35.987340 5812 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:35.988097 5812 heartbeater.cc:511] Master 127.5.105.190:38499 requested a full tablet report, sending...
I20260809 22:53:35.990250 5581 ts_manager.cc:194] Registered new tserver with Master: f9d60226269f41849f171ea50026baa1 (127.5.105.130:45365)
I20260809 22:53:35.990998 5542 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.019395797s
I20260809 22:53:35.992341 5581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60986
I20260809 22:53:36.023797 5581 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:53:36.047317 5581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260809 22:53:36.095713 5812 heartbeater.cc:503] Master 127.5.105.190:38499 was elected leader, sending a full tablet report...
I20260809 22:53:36.103622 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 7894e734d6d44d03a242190f62532b8e (DEFAULT_TABLE table=test-table [id=46caa55f03bd4806b81c8e64c5c702aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:36.105248 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7894e734d6d44d03a242190f62532b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:36.103622 5685 tablet_service.cc:1467] Processing CreateTablet for tablet d665eaf624654bbfa8ec61aeeb4abbdf (DEFAULT_TABLE table=test-table [id=46caa55f03bd4806b81c8e64c5c702aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:36.111393 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d665eaf624654bbfa8ec61aeeb4abbdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:36.112998 5768 tablet_service.cc:1467] Processing CreateTablet for tablet 7894e734d6d44d03a242190f62532b8e (DEFAULT_TABLE table=test-table [id=46caa55f03bd4806b81c8e64c5c702aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:36.114349 5768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7894e734d6d44d03a242190f62532b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:36.120157 5767 tablet_service.cc:1467] Processing CreateTablet for tablet d665eaf624654bbfa8ec61aeeb4abbdf (DEFAULT_TABLE table=test-table [id=46caa55f03bd4806b81c8e64c5c702aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:36.121481 5767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d665eaf624654bbfa8ec61aeeb4abbdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:36.133163 5826 tablet_bootstrap.cc:492] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0: Bootstrap starting.
I20260809 22:53:36.139458 5827 tablet_bootstrap.cc:492] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1: Bootstrap starting.
I20260809 22:53:36.141743 5826 tablet_bootstrap.cc:654] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:36.148476 5827 tablet_bootstrap.cc:654] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:36.148648 5826 tablet_bootstrap.cc:492] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0: No bootstrap required, opened a new log
I20260809 22:53:36.149109 5826 ts_tablet_manager.cc:1397] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.013s
I20260809 22:53:36.152242 5826 raft_consensus.cc:348] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } }
I20260809 22:53:36.153021 5826 raft_consensus.cc:729] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797286e6cdb44f17a901a8d4babbb9c0, State: Initialized, Role: FOLLOWER
I20260809 22:53:36.153635 5826 consensus_queue.cc:260] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } }
I20260809 22:53:36.155896 5827 tablet_bootstrap.cc:492] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1: No bootstrap required, opened a new log
I20260809 22:53:36.156358 5827 ts_tablet_manager.cc:1397] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.014s
I20260809 22:53:36.159305 5826 ts_tablet_manager.cc:1428] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0: Time spent starting tablet: real 0.010s user 0.003s sys 0.004s
I20260809 22:53:36.158963 5827 raft_consensus.cc:348] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } }
I20260809 22:53:36.159857 5827 raft_consensus.cc:729] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d60226269f41849f171ea50026baa1, State: Initialized, Role: FOLLOWER
I20260809 22:53:36.160287 5826 tablet_bootstrap.cc:492] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0: Bootstrap starting.
I20260809 22:53:36.160526 5827 consensus_queue.cc:260] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } }
I20260809 22:53:36.170864 5827 ts_tablet_manager.cc:1428] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260809 22:53:36.171573 5826 tablet_bootstrap.cc:654] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:36.173225 5827 tablet_bootstrap.cc:492] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1: Bootstrap starting.
I20260809 22:53:36.181088 5827 tablet_bootstrap.cc:654] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:36.185086 5826 tablet_bootstrap.cc:492] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0: No bootstrap required, opened a new log
I20260809 22:53:36.185497 5826 ts_tablet_manager.cc:1397] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.005s
I20260809 22:53:36.187403 5826 raft_consensus.cc:348] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } } peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } }
I20260809 22:53:36.187942 5826 raft_consensus.cc:729] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797286e6cdb44f17a901a8d4babbb9c0, State: Initialized, Role: FOLLOWER
I20260809 22:53:36.188395 5826 consensus_queue.cc:260] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } } peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } }
I20260809 22:53:36.190215 5826 ts_tablet_manager.cc:1428] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260809 22:53:36.192973 5827 tablet_bootstrap.cc:492] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1: No bootstrap required, opened a new log
I20260809 22:53:36.193434 5827 ts_tablet_manager.cc:1397] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.001s
I20260809 22:53:36.195977 5827 raft_consensus.cc:348] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } } peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } }
I20260809 22:53:36.196630 5827 raft_consensus.cc:729] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d60226269f41849f171ea50026baa1, State: Initialized, Role: FOLLOWER
I20260809 22:53:36.197178 5827 consensus_queue.cc:260] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } } peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } }
I20260809 22:53:36.199424 5827 ts_tablet_manager.cc:1428] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20260809 22:53:36.223345 5813 tablet.cc:2369] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:53:36.226670 5542 ts_tablet_manager-itest.cc:367] Electing peer 2...
I20260809 22:53:36.226979 5542 raft_consensus.cc:412] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [term 0 FOLLOWER]: Emulating election...
I20260809 22:53:36.227229 5542 raft_consensus.cc:3037] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:36.231546 5542 raft_consensus.cc:686] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [term 1 LEADER]: Becoming Leader. State: Replica: f9d60226269f41849f171ea50026baa1, State: Running, Role: LEADER
I20260809 22:53:36.232167 5542 consensus_queue.cc:237] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } } peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } }
I20260809 22:53:36.235926 5542 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260809 22:53:36.242662 5580 catalog_manager.cc:5495] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d60226269f41849f171ea50026baa1 (127.5.105.130). New cstate: current_term: 1 leader_uuid: "f9d60226269f41849f171ea50026baa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } health_report { overall_health: HEALTHY } } }
I20260809 22:53:36.269450 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260809 22:53:36.373700 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260809 22:53:36.577605 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260809 22:53:36.805985 5698 raft_consensus.cc:3037] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:36.815204 5837 consensus_queue.cc:1035] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:53:36.881543 5542 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20260809 22:53:36.881847 5542 raft_consensus.cc:412] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 0 FOLLOWER]: Emulating election...
I20260809 22:53:36.882128 5542 raft_consensus.cc:3037] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:36.887969 5542 raft_consensus.cc:686] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 1 LEADER]: Becoming Leader. State: Replica: f9d60226269f41849f171ea50026baa1, State: Running, Role: LEADER
I20260809 22:53:36.888625 5542 consensus_queue.cc:237] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } }
I20260809 22:53:36.891152 5542 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260809 22:53:36.895000 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260809 22:53:36.895145 5581 catalog_manager.cc:5495] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d60226269f41849f171ea50026baa1 (127.5.105.130). New cstate: current_term: 1 leader_uuid: "f9d60226269f41849f171ea50026baa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } health_report { overall_health: UNKNOWN } } }
I20260809 22:53:36.998733 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260809 22:53:37.202495 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260809 22:53:37.428442 5698 raft_consensus.cc:3037] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:37.435974 5852 consensus_queue.cc:1035] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:53:37.706435 5542 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20260809 22:53:37.706743 5542 raft_consensus.cc:412] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 1 LEADER]: Emulating election...
I20260809 22:53:37.706991 5542 raft_consensus.cc:3032] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 22:53:37.707216 5542 raft_consensus.cc:729] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9d60226269f41849f171ea50026baa1, State: Running, Role: LEADER
I20260809 22:53:37.707692 5542 consensus_queue.cc:260] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } }
I20260809 22:53:37.708474 5542 raft_consensus.cc:3037] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:37.712534 5542 raft_consensus.cc:686] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 2 LEADER]: Becoming Leader. State: Replica: f9d60226269f41849f171ea50026baa1, State: Running, Role: LEADER
I20260809 22:53:37.713106 5542 consensus_queue.cc:237] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } }
I20260809 22:53:37.714517 5542 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260809 22:53:37.718475 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20260809 22:53:37.718207 5580 catalog_manager.cc:5495] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "f9d60226269f41849f171ea50026baa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } health_report { overall_health: UNKNOWN } } }
I20260809 22:53:37.822160 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20260809 22:53:38.026216 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20260809 22:53:38.191764 5698 raft_consensus.cc:3037] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:38.196182 5698 raft_consensus.cc:1261] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 2 FOLLOWER]: Refusing update from remote peer f9d60226269f41849f171ea50026baa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 22:53:38.197295 5855 consensus_queue.cc:1035] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 22:53:38.329854 5542 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260809 22:53:38.330134 5542 raft_consensus.cc:412] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0 [term 1 FOLLOWER]: Emulating election...
I20260809 22:53:38.330353 5542 raft_consensus.cc:3037] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:38.334108 5542 raft_consensus.cc:686] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0 [term 2 LEADER]: Becoming Leader. State: Replica: 797286e6cdb44f17a901a8d4babbb9c0, State: Running, Role: LEADER
I20260809 22:53:38.334729 5542 consensus_queue.cc:237] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } } peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } }
I20260809 22:53:38.336763 5542 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260809 22:53:38.340775 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20260809 22:53:38.340719 5580 catalog_manager.cc:5495] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0 reported cstate change: term changed from 1 to 2, leader changed from f9d60226269f41849f171ea50026baa1 (127.5.105.130) to 797286e6cdb44f17a901a8d4babbb9c0 (127.5.105.129). New cstate: current_term: 2 leader_uuid: "797286e6cdb44f17a901a8d4babbb9c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } health_report { overall_health: UNKNOWN } } }
I20260809 22:53:38.389238 5698 raft_consensus.cc:1226] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0 [term 2 LEADER]: Rejecting Update request from peer f9d60226269f41849f171ea50026baa1 for earlier term 1. Current term is 2. Ops: []
I20260809 22:53:38.390574 5862 consensus_queue.cc:1046] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 22:53:38.391335 5862 consensus_queue.cc:1225] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [LEADER]: Peer 797286e6cdb44f17a901a8d4babbb9c0 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260809 22:53:38.391729 5855 raft_consensus.cc:3032] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 22:53:38.392140 5855 raft_consensus.cc:729] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9d60226269f41849f171ea50026baa1, State: Running, Role: LEADER
I20260809 22:53:38.392946 5855 consensus_queue.cc:260] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } } peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } }
I20260809 22:53:38.393838 5855 raft_consensus.cc:3037] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:38.444383 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20260809 22:53:38.648600 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20260809 22:53:38.891981 5780 raft_consensus.cc:1261] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [term 2 FOLLOWER]: Refusing update from remote peer 797286e6cdb44f17a901a8d4babbb9c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 22:53:38.893600 5870 consensus_queue.cc:1035] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 22:53:39.152565 5542 ts_tablet_manager-itest.cc:367] Electing peer 1...
I20260809 22:53:39.152799 5542 raft_consensus.cc:412] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 2 FOLLOWER]: Emulating election...
I20260809 22:53:39.153008 5542 raft_consensus.cc:3037] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:39.157009 5542 raft_consensus.cc:686] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 3 LEADER]: Becoming Leader. State: Replica: 797286e6cdb44f17a901a8d4babbb9c0, State: Running, Role: LEADER
I20260809 22:53:39.157683 5542 consensus_queue.cc:237] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } }
I20260809 22:53:39.159371 5542 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260809 22:53:39.163084 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20260809 22:53:39.162978 5581 catalog_manager.cc:5495] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 reported cstate change: term changed from 2 to 3, leader changed from f9d60226269f41849f171ea50026baa1 (127.5.105.130) to 797286e6cdb44f17a901a8d4babbb9c0 (127.5.105.129). New cstate: current_term: 3 leader_uuid: "797286e6cdb44f17a901a8d4babbb9c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } health_report { overall_health: HEALTHY } } }
I20260809 22:53:39.266584 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20260809 22:53:39.470134 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20260809 22:53:39.606174 5698 raft_consensus.cc:1226] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 3 LEADER]: Rejecting Update request from peer f9d60226269f41849f171ea50026baa1 for earlier term 2. Current term is 3. Ops: []
I20260809 22:53:39.607331 5880 consensus_queue.cc:1046] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 22:53:39.608268 5880 consensus_queue.cc:1225] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [LEADER]: Peer 797286e6cdb44f17a901a8d4babbb9c0 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260809 22:53:39.608727 5881 raft_consensus.cc:3032] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 22:53:39.609004 5881 raft_consensus.cc:729] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f9d60226269f41849f171ea50026baa1, State: Running, Role: LEADER
I20260809 22:53:39.609628 5881 consensus_queue.cc:260] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } }
I20260809 22:53:39.610332 5881 raft_consensus.cc:3037] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:39.676447 5780 raft_consensus.cc:1261] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 3 FOLLOWER]: Refusing update from remote peer 797286e6cdb44f17a901a8d4babbb9c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 22:53:39.677577 5870 consensus_queue.cc:1035] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 22:53:39.774452 5542 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20260809 22:53:39.774711 5542 raft_consensus.cc:412] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 3 FOLLOWER]: Emulating election...
I20260809 22:53:39.774969 5542 raft_consensus.cc:3037] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 3 FOLLOWER]: Advancing to term 4
I20260809 22:53:39.778960 5542 raft_consensus.cc:686] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 4 LEADER]: Becoming Leader. State: Replica: f9d60226269f41849f171ea50026baa1, State: Running, Role: LEADER
I20260809 22:53:39.779572 5542 consensus_queue.cc:237] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } }
I20260809 22:53:39.780834 5542 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260809 22:53:39.784183 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20260809 22:53:39.784482 5580 catalog_manager.cc:5495] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 reported cstate change: term changed from 3 to 4, leader changed from 797286e6cdb44f17a901a8d4babbb9c0 (127.5.105.129) to f9d60226269f41849f171ea50026baa1 (127.5.105.130). New cstate: current_term: 4 leader_uuid: "f9d60226269f41849f171ea50026baa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } health_report { overall_health: UNKNOWN } } }
I20260809 22:53:39.887518 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20260809 22:53:40.090961 5542 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20260809 22:53:40.172967 5780 raft_consensus.cc:1226] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 4 LEADER]: Rejecting Update request from peer 797286e6cdb44f17a901a8d4babbb9c0 for earlier term 3. Current term is 4. Ops: []
I20260809 22:53:40.174266 5870 consensus_queue.cc:1046] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 22:53:40.174880 5870 consensus_queue.cc:1225] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [LEADER]: Peer f9d60226269f41849f171ea50026baa1 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260809 22:53:40.175221 5883 raft_consensus.cc:3032] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 22:53:40.175516 5883 raft_consensus.cc:729] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 797286e6cdb44f17a901a8d4babbb9c0, State: Running, Role: LEADER
I20260809 22:53:40.176011 5883 consensus_queue.cc:260] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d60226269f41849f171ea50026baa1" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 45365 } } peers { permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 } }
I20260809 22:53:40.176712 5883 raft_consensus.cc:3037] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 3 FOLLOWER]: Advancing to term 4
I20260809 22:53:40.371210 5698 raft_consensus.cc:1261] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 4 FOLLOWER]: Refusing update from remote peer f9d60226269f41849f171ea50026baa1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 22:53:40.372604 5892 consensus_queue.cc:1035] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "797286e6cdb44f17a901a8d4babbb9c0" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 22:53:40.599290 5542 tablet_server.cc:178] TabletServer@127.5.105.129:0 shutting down...
I20260809 22:53:40.618170 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:53:40.618786 5542 tablet_replica.cc:330] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0: stopping tablet replica
I20260809 22:53:40.619349 5542 raft_consensus.cc:2229] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0 [term 2 LEADER]: Raft consensus shutting down.
I20260809 22:53:40.619951 5542 raft_consensus.cc:2258] T d665eaf624654bbfa8ec61aeeb4abbdf P 797286e6cdb44f17a901a8d4babbb9c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:40.621869 5542 tablet_replica.cc:330] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0: stopping tablet replica
I20260809 22:53:40.622285 5542 raft_consensus.cc:2229] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260809 22:53:40.622714 5542 raft_consensus.cc:2258] T 7894e734d6d44d03a242190f62532b8e P 797286e6cdb44f17a901a8d4babbb9c0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:40.722558 5542 tablet_server.cc:194] TabletServer@127.5.105.129:0 shutdown complete.
I20260809 22:53:40.731666 5542 tablet_server.cc:178] TabletServer@127.5.105.130:0 shutting down...
I20260809 22:53:40.748454 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:53:40.749033 5542 tablet_replica.cc:330] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1: stopping tablet replica
I20260809 22:53:40.749562 5542 raft_consensus.cc:2229] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 22:53:40.749917 5542 raft_consensus.cc:2258] T d665eaf624654bbfa8ec61aeeb4abbdf P f9d60226269f41849f171ea50026baa1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:40.751327 5542 tablet_replica.cc:330] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1: stopping tablet replica
I20260809 22:53:40.751693 5542 raft_consensus.cc:2229] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 4 LEADER]: Raft consensus shutting down.
I20260809 22:53:40.752180 5542 raft_consensus.cc:2258] T 7894e734d6d44d03a242190f62532b8e P f9d60226269f41849f171ea50026baa1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:40.760826 5542 tablet_server.cc:194] TabletServer@127.5.105.130:0 shutdown complete.
I20260809 22:53:40.769057 5542 master.cc:537] Master@127.5.105.190:38499 shutting down...
I20260809 22:53:41.013629 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:53:41.014138 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:41.014513 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P c87e01b30e244dff8486756002730b00: stopping tablet replica
I20260809 22:53:41.032769 5542 master.cc:552] Master@127.5.105.190:38499 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (7267 ms)
[ RUN ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20260809 22:53:41.060324 5542 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.5.105.190:39717
I20260809 22:53:41.061348 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:41.065979 5901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:41.067237 5902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:41.069499 5904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:41.069612 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:41.070333 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:41.070591 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:41.070776 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316021070747 us; error 0 us; skew 500 ppm
I20260809 22:53:41.071290 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:41.078446 5542 webserver.cc:460] Webserver started at http://127.5.105.190:42731/ using document root <none> and password file <none>
I20260809 22:53:41.078956 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:41.079142 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:41.079453 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:41.080499 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/master-0-root/instance:
uuid: "d0a4bc99f3ce4ad9bb2c3b7d34d92008"
format_stamp: "Formatted at 2026-08-09 22:53:41 on dist-test-slave-gp6n"
I20260809 22:53:41.084748 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 22:53:41.087956 5909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:41.088732 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 22:53:41.089020 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/master-0-root
uuid: "d0a4bc99f3ce4ad9bb2c3b7d34d92008"
format_stamp: "Formatted at 2026-08-09 22:53:41 on dist-test-slave-gp6n"
I20260809 22:53:41.089318 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:41.106539 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:41.107555 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:41.143277 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.190:39717
I20260809 22:53:41.143406 5964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.190:39717 every 8 connection(s)
I20260809 22:53:41.146999 5965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:41.156580 5965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008: Bootstrap starting.
I20260809 22:53:41.160785 5965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:41.164587 5965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008: No bootstrap required, opened a new log
I20260809 22:53:41.166334 5965 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a4bc99f3ce4ad9bb2c3b7d34d92008" member_type: VOTER }
I20260809 22:53:41.166806 5965 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:41.167057 5965 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a4bc99f3ce4ad9bb2c3b7d34d92008, State: Initialized, Role: FOLLOWER
I20260809 22:53:41.167580 5965 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a4bc99f3ce4ad9bb2c3b7d34d92008" member_type: VOTER }
I20260809 22:53:41.168047 5965 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:53:41.168283 5965 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:53:41.168552 5965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:41.172190 5965 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a4bc99f3ce4ad9bb2c3b7d34d92008" member_type: VOTER }
I20260809 22:53:41.172690 5965 leader_election.cc:302] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0a4bc99f3ce4ad9bb2c3b7d34d92008; no voters:
I20260809 22:53:41.173732 5965 leader_election.cc:288] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:53:41.174134 5969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:53:41.175482 5969 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [term 1 LEADER]: Becoming Leader. State: Replica: d0a4bc99f3ce4ad9bb2c3b7d34d92008, State: Running, Role: LEADER
I20260809 22:53:41.176054 5969 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a4bc99f3ce4ad9bb2c3b7d34d92008" member_type: VOTER }
I20260809 22:53:41.176558 5965 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:53:41.178968 5971 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0a4bc99f3ce4ad9bb2c3b7d34d92008. Latest consensus state: current_term: 1 leader_uuid: "d0a4bc99f3ce4ad9bb2c3b7d34d92008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a4bc99f3ce4ad9bb2c3b7d34d92008" member_type: VOTER } }
I20260809 22:53:41.179050 5970 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0a4bc99f3ce4ad9bb2c3b7d34d92008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a4bc99f3ce4ad9bb2c3b7d34d92008" member_type: VOTER } }
I20260809 22:53:41.179633 5971 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:41.179886 5970 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:41.182614 5977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:53:41.187206 5977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:53:41.191625 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 22:53:41.195880 5977 catalog_manager.cc:1347] Generated new cluster ID: 2ede75f951c242588fd3f50ad619804b
I20260809 22:53:41.196147 5977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:53:41.218185 5977 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:53:41.219687 5977 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:53:41.234745 5977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008: Generated new TSK 0
I20260809 22:53:41.235355 5977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:53:41.257845 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:41.263329 5988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:41.264655 5989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:41.267428 5542 server_base.cc:1034] running on GCE node
W20260809 22:53:41.267447 5991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:41.268571 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:41.268748 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:41.268884 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316021268876 us; error 0 us; skew 500 ppm
I20260809 22:53:41.269536 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:41.272015 5542 webserver.cc:460] Webserver started at http://127.5.105.129:32911/ using document root <none> and password file <none>
I20260809 22:53:41.272433 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:41.272583 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:41.272840 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:41.273844 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/ts-0-root/instance:
uuid: "3d7a86c5dcda4edba5b80c059cf11ef1"
format_stamp: "Formatted at 2026-08-09 22:53:41 on dist-test-slave-gp6n"
I20260809 22:53:41.278249 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 22:53:41.281292 5997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:41.282032 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 22:53:41.282258 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "3d7a86c5dcda4edba5b80c059cf11ef1"
format_stamp: "Formatted at 2026-08-09 22:53:41 on dist-test-slave-gp6n"
I20260809 22:53:41.282568 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:41.301481 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:41.302587 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:41.303900 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:53:41.306023 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:53:41.306190 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:41.306515 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:53:41.306653 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:41.342629 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:46805
I20260809 22:53:41.342710 6067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:46805 every 8 connection(s)
I20260809 22:53:41.346971 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:41.353809 6072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:41.355409 6073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:41.358260 6075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:41.359037 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:41.359722 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:41.359994 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:41.360188 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316021360171 us; error 0 us; skew 500 ppm
I20260809 22:53:41.360786 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:41.361212 6068 heartbeater.cc:347] Connected to a master server at 127.5.105.190:39717
I20260809 22:53:41.361595 6068 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:41.362494 6068 heartbeater.cc:511] Master 127.5.105.190:39717 requested a full tablet report, sending...
I20260809 22:53:41.363713 5542 webserver.cc:460] Webserver started at http://127.5.105.130:38419/ using document root <none> and password file <none>
I20260809 22:53:41.364274 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:41.364459 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:41.364629 5926 ts_manager.cc:194] Registered new tserver with Master: 3d7a86c5dcda4edba5b80c059cf11ef1 (127.5.105.129:46805)
I20260809 22:53:41.364732 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:41.365837 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/ts-1-root/instance:
uuid: "99f1c4d7458b4d1cbd576c4e174b62cb"
format_stamp: "Formatted at 2026-08-09 22:53:41 on dist-test-slave-gp6n"
I20260809 22:53:41.366591 5926 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40132
I20260809 22:53:41.370867 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 22:53:41.373931 6081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:41.374706 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260809 22:53:41.374975 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/ts-1-root
uuid: "99f1c4d7458b4d1cbd576c4e174b62cb"
format_stamp: "Formatted at 2026-08-09 22:53:41 on dist-test-slave-gp6n"
I20260809 22:53:41.375259 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786316013627029-5542-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:41.385423 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:41.386588 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:41.387862 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:53:41.389991 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:53:41.390213 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:41.390484 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:53:41.390651 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:41.426218 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.130:40899
I20260809 22:53:41.426318 6150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.130:40899 every 8 connection(s)
I20260809 22:53:41.440207 6151 heartbeater.cc:347] Connected to a master server at 127.5.105.190:39717
I20260809 22:53:41.440559 6151 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:41.441200 6151 heartbeater.cc:511] Master 127.5.105.190:39717 requested a full tablet report, sending...
I20260809 22:53:41.443068 5926 ts_manager.cc:194] Registered new tserver with Master: 99f1c4d7458b4d1cbd576c4e174b62cb (127.5.105.130:40899)
I20260809 22:53:41.443502 5542 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.014255766s
I20260809 22:53:41.444541 5926 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43240
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:421: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20260809 22:53:41.464648 5542 tablet_server.cc:178] TabletServer@127.5.105.129:0 shutting down...
I20260809 22:53:41.473239 6068 heartbeater.cc:503] Master 127.5.105.190:39717 was elected leader, sending a full tablet report...
I20260809 22:53:41.489614 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:53:41.505023 5542 tablet_server.cc:194] TabletServer@127.5.105.129:0 shutdown complete.
I20260809 22:53:41.511745 5542 tablet_server.cc:178] TabletServer@127.5.105.130:0 shutting down...
I20260809 22:53:41.526870 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:53:41.542151 5542 tablet_server.cc:194] TabletServer@127.5.105.130:0 shutdown complete.
I20260809 22:53:41.548497 5542 master.cc:537] Master@127.5.105.190:39717 shutting down...
I20260809 22:53:41.561506 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:53:41.561991 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:41.562392 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P d0a4bc99f3ce4ad9bb2c3b7d34d92008: stopping tablet replica
I20260809 22:53:41.580155 5542 master.cc:552] Master@127.5.105.190:39717 shutdown complete.
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (540 ms)
[ RUN ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20260809 22:53:41.600564 5542 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.5.105.190:35847
I20260809 22:53:41.601440 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:41.606009 6166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:41.607066 6167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:41.608880 6169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:41.609714 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:41.610309 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:41.610598 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:41.610762 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316021610751 us; error 0 us; skew 500 ppm
I20260809 22:53:41.611275 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:41.613605 5542 webserver.cc:460] Webserver started at http://127.5.105.190:45791/ using document root <none> and password file <none>
I20260809 22:53:41.614060 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:41.614235 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:41.614584 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:41.615592 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/master-0-root/instance:
uuid: "b03752a531174b468cb9fb5bc9c3b6ce"
format_stamp: "Formatted at 2026-08-09 22:53:41 on dist-test-slave-gp6n"
I20260809 22:53:41.619911 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 22:53:41.622962 6174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:41.623642 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 22:53:41.623896 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/master-0-root
uuid: "b03752a531174b468cb9fb5bc9c3b6ce"
format_stamp: "Formatted at 2026-08-09 22:53:41 on dist-test-slave-gp6n"
I20260809 22:53:41.624148 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:41.639845 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:41.640754 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:41.674737 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.190:35847
I20260809 22:53:41.674854 6230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.190:35847 every 8 connection(s)
I20260809 22:53:41.678332 6231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:41.687265 6231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce: Bootstrap starting.
I20260809 22:53:41.691002 6231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:41.694681 6231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce: No bootstrap required, opened a new log
I20260809 22:53:41.696342 6231 raft_consensus.cc:348] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03752a531174b468cb9fb5bc9c3b6ce" member_type: VOTER }
I20260809 22:53:41.696723 6231 raft_consensus.cc:374] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:41.697027 6231 raft_consensus.cc:729] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03752a531174b468cb9fb5bc9c3b6ce, State: Initialized, Role: FOLLOWER
I20260809 22:53:41.697608 6231 consensus_queue.cc:260] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03752a531174b468cb9fb5bc9c3b6ce" member_type: VOTER }
I20260809 22:53:41.698107 6231 raft_consensus.cc:388] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:53:41.698393 6231 raft_consensus.cc:482] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:53:41.698690 6231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:41.702360 6231 raft_consensus.cc:504] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03752a531174b468cb9fb5bc9c3b6ce" member_type: VOTER }
I20260809 22:53:41.702921 6231 leader_election.cc:302] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b03752a531174b468cb9fb5bc9c3b6ce; no voters:
I20260809 22:53:41.703946 6231 leader_election.cc:288] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:53:41.704239 6234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:53:41.705580 6234 raft_consensus.cc:686] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [term 1 LEADER]: Becoming Leader. State: Replica: b03752a531174b468cb9fb5bc9c3b6ce, State: Running, Role: LEADER
I20260809 22:53:41.706218 6234 consensus_queue.cc:237] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03752a531174b468cb9fb5bc9c3b6ce" member_type: VOTER }
I20260809 22:53:41.706846 6231 sys_catalog.cc:564] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:53:41.708843 6236 sys_catalog.cc:455] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b03752a531174b468cb9fb5bc9c3b6ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03752a531174b468cb9fb5bc9c3b6ce" member_type: VOTER } }
I20260809 22:53:41.708843 6237 sys_catalog.cc:455] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader b03752a531174b468cb9fb5bc9c3b6ce. Latest consensus state: current_term: 1 leader_uuid: "b03752a531174b468cb9fb5bc9c3b6ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03752a531174b468cb9fb5bc9c3b6ce" member_type: VOTER } }
I20260809 22:53:41.709484 6236 sys_catalog.cc:458] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:41.709620 6237 sys_catalog.cc:458] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:41.712791 6240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:53:41.720393 6240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:53:41.721379 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 22:53:41.728672 6240 catalog_manager.cc:1347] Generated new cluster ID: 71b1f4344d2b4e87977c289bd8df473b
I20260809 22:53:41.728966 6240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:53:41.759895 6240 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:53:41.761407 6240 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:53:41.773491 6240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce: Generated new TSK 0
I20260809 22:53:41.774142 6240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:53:41.787673 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:41.793063 6254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:41.794211 6255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:41.795926 5542 server_base.cc:1034] running on GCE node
W20260809 22:53:41.795882 6257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:41.796919 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:41.797087 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:41.797307 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316021797291 us; error 0 us; skew 500 ppm
I20260809 22:53:41.798089 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:41.800406 5542 webserver.cc:460] Webserver started at http://127.5.105.129:35905/ using document root <none> and password file <none>
I20260809 22:53:41.800817 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:41.800968 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:41.801158 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:41.802101 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/ts-0-root/instance:
uuid: "177fd8226df34bbc91582c419133b331"
format_stamp: "Formatted at 2026-08-09 22:53:41 on dist-test-slave-gp6n"
I20260809 22:53:41.806171 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260809 22:53:41.809562 6262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:41.810297 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 22:53:41.810635 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "177fd8226df34bbc91582c419133b331"
format_stamp: "Formatted at 2026-08-09 22:53:41 on dist-test-slave-gp6n"
I20260809 22:53:41.810889 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:41.822455 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:41.823436 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:41.824738 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:53:41.826913 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:53:41.827082 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:41.827344 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:53:41.827497 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:41.867715 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:37207
I20260809 22:53:41.867820 6333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:37207 every 8 connection(s)
I20260809 22:53:41.872313 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:41.879026 6339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:41.880748 6340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:41.882256 5542 server_base.cc:1034] running on GCE node
W20260809 22:53:41.882649 6342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:41.883531 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:41.883709 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:41.883880 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316021883869 us; error 0 us; skew 500 ppm
I20260809 22:53:41.884591 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:41.887354 5542 webserver.cc:460] Webserver started at http://127.5.105.130:33963/ using document root <none> and password file <none>
I20260809 22:53:41.887900 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:41.888103 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:41.888383 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:41.888633 6334 heartbeater.cc:347] Connected to a master server at 127.5.105.190:35847
I20260809 22:53:41.888995 6334 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:41.889768 6334 heartbeater.cc:511] Master 127.5.105.190:35847 requested a full tablet report, sending...
I20260809 22:53:41.889768 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/ts-1-root/instance:
uuid: "d6caf142d6a54ea3999b50d681336a40"
format_stamp: "Formatted at 2026-08-09 22:53:41 on dist-test-slave-gp6n"
I20260809 22:53:41.891820 6192 ts_manager.cc:194] Registered new tserver with Master: 177fd8226df34bbc91582c419133b331 (127.5.105.129:37207)
I20260809 22:53:41.893630 6192 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41062
I20260809 22:53:41.894533 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260809 22:53:41.897715 6347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:41.898540 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 22:53:41.898824 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/ts-1-root
uuid: "d6caf142d6a54ea3999b50d681336a40"
format_stamp: "Formatted at 2026-08-09 22:53:41 on dist-test-slave-gp6n"
I20260809 22:53:41.899087 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786316013627029-5542-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:41.910696 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:41.911646 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:41.912971 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:53:41.915081 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:53:41.915292 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:41.915515 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:53:41.915673 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:41.952126 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.130:34439
I20260809 22:53:41.952211 6417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.130:34439 every 8 connection(s)
I20260809 22:53:41.965464 6419 heartbeater.cc:347] Connected to a master server at 127.5.105.190:35847
I20260809 22:53:41.965839 6419 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:41.966504 6419 heartbeater.cc:511] Master 127.5.105.190:35847 requested a full tablet report, sending...
I20260809 22:53:41.968185 6192 ts_manager.cc:194] Registered new tserver with Master: d6caf142d6a54ea3999b50d681336a40 (127.5.105.130:34439)
I20260809 22:53:41.968350 5542 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.013350144s
I20260809 22:53:41.969708 6192 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41066
I20260809 22:53:41.990904 6192 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:53:41.992399 6192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260809 22:53:41.996271 6334 heartbeater.cc:503] Master 127.5.105.190:35847 was elected leader, sending a full tablet report...
I20260809 22:53:42.041388 6375 tablet_service.cc:1467] Processing CreateTablet for tablet 4370e7247bbb44d7b7703c9dcf3876fa (DEFAULT_TABLE table=test-table [id=af70f883d27e4abbb00c6b04671104b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:42.041427 6293 tablet_service.cc:1467] Processing CreateTablet for tablet 4370e7247bbb44d7b7703c9dcf3876fa (DEFAULT_TABLE table=test-table [id=af70f883d27e4abbb00c6b04671104b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:42.042552 6375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4370e7247bbb44d7b7703c9dcf3876fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:42.042579 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4370e7247bbb44d7b7703c9dcf3876fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:42.054821 6433 tablet_bootstrap.cc:492] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331: Bootstrap starting.
I20260809 22:53:42.055197 6435 tablet_bootstrap.cc:492] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40: Bootstrap starting.
I20260809 22:53:42.058861 6433 tablet_bootstrap.cc:654] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:42.059320 6435 tablet_bootstrap.cc:654] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:42.063443 6433 tablet_bootstrap.cc:492] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331: No bootstrap required, opened a new log
I20260809 22:53:42.063568 6435 tablet_bootstrap.cc:492] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40: No bootstrap required, opened a new log
I20260809 22:53:42.063866 6433 ts_tablet_manager.cc:1397] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 22:53:42.063951 6435 ts_tablet_manager.cc:1397] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260809 22:53:42.066303 6433 raft_consensus.cc:348] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177fd8226df34bbc91582c419133b331" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37207 } } peers { permanent_uuid: "d6caf142d6a54ea3999b50d681336a40" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 34439 } }
I20260809 22:53:42.066448 6435 raft_consensus.cc:348] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177fd8226df34bbc91582c419133b331" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37207 } } peers { permanent_uuid: "d6caf142d6a54ea3999b50d681336a40" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 34439 } }
I20260809 22:53:42.067034 6435 raft_consensus.cc:729] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6caf142d6a54ea3999b50d681336a40, State: Initialized, Role: FOLLOWER
I20260809 22:53:42.067008 6433 raft_consensus.cc:729] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 177fd8226df34bbc91582c419133b331, State: Initialized, Role: FOLLOWER
I20260809 22:53:42.067751 6433 consensus_queue.cc:260] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177fd8226df34bbc91582c419133b331" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37207 } } peers { permanent_uuid: "d6caf142d6a54ea3999b50d681336a40" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 34439 } }
I20260809 22:53:42.067751 6435 consensus_queue.cc:260] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177fd8226df34bbc91582c419133b331" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37207 } } peers { permanent_uuid: "d6caf142d6a54ea3999b50d681336a40" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 34439 } }
I20260809 22:53:42.071492 6419 heartbeater.cc:503] Master 127.5.105.190:35847 was elected leader, sending a full tablet report...
I20260809 22:53:42.071617 6435 ts_tablet_manager.cc:1428] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260809 22:53:42.071713 6433 ts_tablet_manager.cc:1428] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331: Time spent starting tablet: real 0.008s user 0.002s sys 0.007s
I20260809 22:53:42.104061 5542 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260809 22:53:42.104312 5542 raft_consensus.cc:412] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 0 FOLLOWER]: Emulating election...
I20260809 22:53:42.104565 5542 raft_consensus.cc:3037] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:42.108645 5542 raft_consensus.cc:686] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 1 LEADER]: Becoming Leader. State: Replica: 177fd8226df34bbc91582c419133b331, State: Running, Role: LEADER
W20260809 22:53:42.108995 6334 heartbeater.cc:644] Failed to heartbeat to 127.5.105.190:35847 (0 consecutive failures): IO error: failing all heartbeats for tests
I20260809 22:53:42.109378 5542 consensus_queue.cc:237] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177fd8226df34bbc91582c419133b331" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37207 } } peers { permanent_uuid: "d6caf142d6a54ea3999b50d681336a40" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 34439 } }
I20260809 22:53:42.112079 5542 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260809 22:53:42.133585 5542 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20260809 22:53:42.237584 5542 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20260809 22:53:42.441614 5542 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20260809 22:53:42.577085 6389 raft_consensus.cc:3037] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:42.582998 6439 consensus_queue.cc:1035] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6caf142d6a54ea3999b50d681336a40" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 34439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:53:42.746906 5542 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260809 22:53:42.747116 5542 raft_consensus.cc:412] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 1 LEADER]: Emulating election...
I20260809 22:53:42.747279 5542 raft_consensus.cc:3032] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 22:53:42.747421 5542 raft_consensus.cc:729] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 177fd8226df34bbc91582c419133b331, State: Running, Role: LEADER
I20260809 22:53:42.747779 5542 consensus_queue.cc:260] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177fd8226df34bbc91582c419133b331" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37207 } } peers { permanent_uuid: "d6caf142d6a54ea3999b50d681336a40" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 34439 } }
I20260809 22:53:42.748404 5542 raft_consensus.cc:3037] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:42.752172 5542 raft_consensus.cc:686] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 2 LEADER]: Becoming Leader. State: Replica: 177fd8226df34bbc91582c419133b331, State: Running, Role: LEADER
I20260809 22:53:42.752673 5542 consensus_queue.cc:237] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177fd8226df34bbc91582c419133b331" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37207 } } peers { permanent_uuid: "d6caf142d6a54ea3999b50d681336a40" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 34439 } }
I20260809 22:53:42.753984 5542 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260809 22:53:42.757098 5542 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20260809 22:53:42.860893 5542 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20260809 22:53:43.064937 5542 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20260809 22:53:43.307487 6389 raft_consensus.cc:3037] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:43.312168 6389 raft_consensus.cc:1261] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40 [term 2 FOLLOWER]: Refusing update from remote peer 177fd8226df34bbc91582c419133b331: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 22:53:43.313446 6450 consensus_queue.cc:1035] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6caf142d6a54ea3999b50d681336a40" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 34439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 22:53:43.369910 5542 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260809 22:53:43.370137 5542 raft_consensus.cc:412] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 2 LEADER]: Emulating election...
I20260809 22:53:43.370304 5542 raft_consensus.cc:3032] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 22:53:43.370492 5542 raft_consensus.cc:729] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 177fd8226df34bbc91582c419133b331, State: Running, Role: LEADER
I20260809 22:53:43.370829 5542 consensus_queue.cc:260] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177fd8226df34bbc91582c419133b331" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37207 } } peers { permanent_uuid: "d6caf142d6a54ea3999b50d681336a40" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 34439 } }
I20260809 22:53:43.371461 5542 raft_consensus.cc:3037] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:43.375531 5542 raft_consensus.cc:686] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 3 LEADER]: Becoming Leader. State: Replica: 177fd8226df34bbc91582c419133b331, State: Running, Role: LEADER
I20260809 22:53:43.376082 5542 consensus_queue.cc:237] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "177fd8226df34bbc91582c419133b331" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37207 } } peers { permanent_uuid: "d6caf142d6a54ea3999b50d681336a40" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 34439 } }
I20260809 22:53:43.377240 5542 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260809 22:53:43.380458 5542 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20260809 22:53:43.484496 5542 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20260809 22:53:43.688267 5542 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20260809 22:53:43.861901 6389 raft_consensus.cc:3037] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:43.867980 6389 raft_consensus.cc:1261] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40 [term 3 FOLLOWER]: Refusing update from remote peer 177fd8226df34bbc91582c419133b331: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 22:53:43.869474 6450 consensus_queue.cc:1035] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6caf142d6a54ea3999b50d681336a40" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 34439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 22:53:43.997315 5542 tablet_server.cc:178] TabletServer@127.5.105.129:0 shutting down...
I20260809 22:53:44.023517 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:53:44.024209 5542 tablet_replica.cc:330] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331: stopping tablet replica
I20260809 22:53:44.024791 5542 raft_consensus.cc:2229] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 3 LEADER]: Raft consensus shutting down.
I20260809 22:53:44.025470 5542 raft_consensus.cc:2258] T 4370e7247bbb44d7b7703c9dcf3876fa P 177fd8226df34bbc91582c419133b331 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:44.044953 5542 tablet_server.cc:194] TabletServer@127.5.105.129:0 shutdown complete.
I20260809 22:53:44.054778 5542 tablet_server.cc:178] TabletServer@127.5.105.130:0 shutting down...
I20260809 22:53:44.074538 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:53:44.075096 5542 tablet_replica.cc:330] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40: stopping tablet replica
I20260809 22:53:44.075587 5542 raft_consensus.cc:2229] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 22:53:44.075934 5542 raft_consensus.cc:2258] T 4370e7247bbb44d7b7703c9dcf3876fa P d6caf142d6a54ea3999b50d681336a40 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:44.094705 5542 tablet_server.cc:194] TabletServer@127.5.105.130:0 shutdown complete.
I20260809 22:53:44.102852 5542 master.cc:537] Master@127.5.105.190:35847 shutting down...
I20260809 22:53:44.122452 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:53:44.122954 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:44.123214 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P b03752a531174b468cb9fb5bc9c3b6ce: stopping tablet replica
I20260809 22:53:44.140277 5542 master.cc:552] Master@127.5.105.190:35847 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (2569 ms)
[----------] 3 tests from LeadershipChangeReportingTest (10376 ms total)
[----------] 4 tests from TsTabletManagerITest
[ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20260809 22:53:44.170100 5542 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.5.105.190:41455
I20260809 22:53:44.171208 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:44.176612 6467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:44.177498 6468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:44.178813 6470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:44.178961 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:44.179813 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:44.180006 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:44.180115 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316024180107 us; error 0 us; skew 500 ppm
I20260809 22:53:44.180541 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:44.183064 5542 webserver.cc:460] Webserver started at http://127.5.105.190:44129/ using document root <none> and password file <none>
I20260809 22:53:44.183478 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:44.183629 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:44.183853 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:44.185011 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/master-0-root/instance:
uuid: "c44a3856917a4a2b80689e94b4994a69"
format_stamp: "Formatted at 2026-08-09 22:53:44 on dist-test-slave-gp6n"
I20260809 22:53:44.189201 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 22:53:44.192279 6476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:44.192986 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 22:53:44.193203 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/master-0-root
uuid: "c44a3856917a4a2b80689e94b4994a69"
format_stamp: "Formatted at 2026-08-09 22:53:44 on dist-test-slave-gp6n"
I20260809 22:53:44.193424 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:44.218974 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:44.220678 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:44.258781 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.190:41455
I20260809 22:53:44.258908 6534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.190:41455 every 8 connection(s)
I20260809 22:53:44.262598 6535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:44.272122 6535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69: Bootstrap starting.
I20260809 22:53:44.276196 6535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:44.280283 6535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69: No bootstrap required, opened a new log
I20260809 22:53:44.282335 6535 raft_consensus.cc:348] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44a3856917a4a2b80689e94b4994a69" member_type: VOTER }
I20260809 22:53:44.282841 6535 raft_consensus.cc:374] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:44.283041 6535 raft_consensus.cc:729] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c44a3856917a4a2b80689e94b4994a69, State: Initialized, Role: FOLLOWER
I20260809 22:53:44.283615 6535 consensus_queue.cc:260] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44a3856917a4a2b80689e94b4994a69" member_type: VOTER }
I20260809 22:53:44.284073 6535 raft_consensus.cc:388] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:53:44.284283 6535 raft_consensus.cc:482] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:53:44.284499 6535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:44.288695 6535 raft_consensus.cc:504] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44a3856917a4a2b80689e94b4994a69" member_type: VOTER }
I20260809 22:53:44.289176 6535 leader_election.cc:302] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c44a3856917a4a2b80689e94b4994a69; no voters:
I20260809 22:53:44.290254 6535 leader_election.cc:288] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:53:44.290693 6538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:53:44.292062 6538 raft_consensus.cc:686] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [term 1 LEADER]: Becoming Leader. State: Replica: c44a3856917a4a2b80689e94b4994a69, State: Running, Role: LEADER
I20260809 22:53:44.292711 6538 consensus_queue.cc:237] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44a3856917a4a2b80689e94b4994a69" member_type: VOTER }
I20260809 22:53:44.293488 6535 sys_catalog.cc:564] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:53:44.295867 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c44a3856917a4a2b80689e94b4994a69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44a3856917a4a2b80689e94b4994a69" member_type: VOTER } }
I20260809 22:53:44.295843 6540 sys_catalog.cc:455] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c44a3856917a4a2b80689e94b4994a69. Latest consensus state: current_term: 1 leader_uuid: "c44a3856917a4a2b80689e94b4994a69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44a3856917a4a2b80689e94b4994a69" member_type: VOTER } }
I20260809 22:53:44.296554 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:44.296677 6540 sys_catalog.cc:458] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:44.299266 6544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:53:44.305536 6544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:53:44.311183 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 22:53:44.316015 6544 catalog_manager.cc:1347] Generated new cluster ID: 1aac8dd9b11841559e9d2aba226ba284
I20260809 22:53:44.316272 6544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:53:44.333667 6544 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:53:44.335466 6544 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:53:44.349447 6544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69: Generated new TSK 0
I20260809 22:53:44.350102 6544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:53:44.377287 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:44.383374 6556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:44.384500 6557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:44.385716 6559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:44.386274 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:44.387106 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:44.387269 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:44.387385 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316024387377 us; error 0 us; skew 500 ppm
I20260809 22:53:44.387851 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:44.389998 5542 webserver.cc:460] Webserver started at http://127.5.105.129:36645/ using document root <none> and password file <none>
I20260809 22:53:44.390492 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:44.390658 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:44.390851 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:44.391944 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-0-root/instance:
uuid: "4d51333c07db4e658383a195687d5040"
format_stamp: "Formatted at 2026-08-09 22:53:44 on dist-test-slave-gp6n"
I20260809 22:53:44.396598 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 22:53:44.399696 6565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:44.400477 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 22:53:44.400806 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "4d51333c07db4e658383a195687d5040"
format_stamp: "Formatted at 2026-08-09 22:53:44 on dist-test-slave-gp6n"
I20260809 22:53:44.401082 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:44.422925 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:44.424048 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:44.425439 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:53:44.427699 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:53:44.427912 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:44.428162 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:53:44.428339 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:44.467089 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:37879
I20260809 22:53:44.467242 6635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:37879 every 8 connection(s)
I20260809 22:53:44.472059 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:44.479439 6641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:44.482566 6643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:44.484726 6645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:44.485419 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:44.486487 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:44.486769 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:44.486965 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316024486948 us; error 0 us; skew 500 ppm
I20260809 22:53:44.487699 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:44.487926 6636 heartbeater.cc:347] Connected to a master server at 127.5.105.190:41455
I20260809 22:53:44.488365 6636 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:44.489302 6636 heartbeater.cc:511] Master 127.5.105.190:41455 requested a full tablet report, sending...
I20260809 22:53:44.490715 5542 webserver.cc:460] Webserver started at http://127.5.105.130:44573/ using document root <none> and password file <none>
I20260809 22:53:44.491335 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:44.491606 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:44.491726 6495 ts_manager.cc:194] Registered new tserver with Master: 4d51333c07db4e658383a195687d5040 (127.5.105.129:37879)
I20260809 22:53:44.491973 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:44.493741 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-1-root/instance:
uuid: "5e0f6b3e81d743aebc8a126063abaa13"
format_stamp: "Formatted at 2026-08-09 22:53:44 on dist-test-slave-gp6n"
I20260809 22:53:44.494653 6495 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44140
I20260809 22:53:44.499781 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 22:53:44.503490 6651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:44.504220 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 22:53:44.504473 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-1-root
uuid: "5e0f6b3e81d743aebc8a126063abaa13"
format_stamp: "Formatted at 2026-08-09 22:53:44 on dist-test-slave-gp6n"
I20260809 22:53:44.504719 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:44.515643 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:44.516832 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:44.518263 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:53:44.520619 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:53:44.520807 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:44.521116 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:53:44.521279 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:44.565845 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.130:39805
I20260809 22:53:44.565948 6722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.130:39805 every 8 connection(s)
I20260809 22:53:44.571066 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:44.578100 6726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:44.579924 6727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:44.581956 6729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:44.582937 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:44.584131 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:44.584367 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:44.584563 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316024584545 us; error 0 us; skew 500 ppm
I20260809 22:53:44.585003 6723 heartbeater.cc:347] Connected to a master server at 127.5.105.190:41455
I20260809 22:53:44.585304 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:44.585322 6723 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:44.586119 6723 heartbeater.cc:511] Master 127.5.105.190:41455 requested a full tablet report, sending...
I20260809 22:53:44.588443 6495 ts_manager.cc:194] Registered new tserver with Master: 5e0f6b3e81d743aebc8a126063abaa13 (127.5.105.130:39805)
I20260809 22:53:44.588601 5542 webserver.cc:460] Webserver started at http://127.5.105.131:44391/ using document root <none> and password file <none>
I20260809 22:53:44.589251 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:44.589530 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:44.589859 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:44.590785 6495 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44150
I20260809 22:53:44.591531 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-2-root/instance:
uuid: "49d2d72c76f74e32a2d9c918701707a8"
format_stamp: "Formatted at 2026-08-09 22:53:44 on dist-test-slave-gp6n"
I20260809 22:53:44.596432 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 22:53:44.599640 6736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:44.600412 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 22:53:44.600704 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-2-root
uuid: "49d2d72c76f74e32a2d9c918701707a8"
format_stamp: "Formatted at 2026-08-09 22:53:44 on dist-test-slave-gp6n"
I20260809 22:53:44.600999 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786316013627029-5542-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:44.630858 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:44.632185 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:44.633668 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:53:44.636018 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:53:44.636206 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:44.636519 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:53:44.636686 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:44.709961 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.131:45005
I20260809 22:53:44.710062 6808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.131:45005 every 8 connection(s)
I20260809 22:53:44.727751 6809 heartbeater.cc:347] Connected to a master server at 127.5.105.190:41455
I20260809 22:53:44.728111 6809 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:44.728767 6809 heartbeater.cc:511] Master 127.5.105.190:41455 requested a full tablet report, sending...
I20260809 22:53:44.730748 6495 ts_manager.cc:194] Registered new tserver with Master: 49d2d72c76f74e32a2d9c918701707a8 (127.5.105.131:45005)
I20260809 22:53:44.730911 5542 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015995143s
I20260809 22:53:44.733120 6495 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44162
I20260809 22:53:44.757063 6495 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260809 22:53:44.758630 6495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:53:44.801055 6681 tablet_service.cc:1467] Processing CreateTablet for tablet d34e2ca4820e456da441c0fc7d6ded1e (DEFAULT_TABLE table=test-table [id=e32d337e8e084412bca530124ed7eb3f]), partition=
I20260809 22:53:44.802291 6681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34e2ca4820e456da441c0fc7d6ded1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:44.804356 6597 tablet_service.cc:1467] Processing CreateTablet for tablet d34e2ca4820e456da441c0fc7d6ded1e (DEFAULT_TABLE table=test-table [id=e32d337e8e084412bca530124ed7eb3f]), partition=
I20260809 22:53:44.805389 6597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34e2ca4820e456da441c0fc7d6ded1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:44.808231 6767 tablet_service.cc:1467] Processing CreateTablet for tablet d34e2ca4820e456da441c0fc7d6ded1e (DEFAULT_TABLE table=test-table [id=e32d337e8e084412bca530124ed7eb3f]), partition=
I20260809 22:53:44.808964 6767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34e2ca4820e456da441c0fc7d6ded1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:44.818205 6825 tablet_bootstrap.cc:492] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13: Bootstrap starting.
I20260809 22:53:44.822207 6826 tablet_bootstrap.cc:492] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040: Bootstrap starting.
I20260809 22:53:44.825814 6825 tablet_bootstrap.cc:654] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:44.827355 6826 tablet_bootstrap.cc:654] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:44.828610 6828 tablet_bootstrap.cc:492] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8: Bootstrap starting.
I20260809 22:53:44.833204 6825 tablet_bootstrap.cc:492] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13: No bootstrap required, opened a new log
I20260809 22:53:44.833585 6825 ts_tablet_manager.cc:1397] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.003s
I20260809 22:53:44.833902 6826 tablet_bootstrap.cc:492] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040: No bootstrap required, opened a new log
I20260809 22:53:44.834347 6826 ts_tablet_manager.cc:1397] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260809 22:53:44.835704 6828 tablet_bootstrap.cc:654] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:44.836447 6825 raft_consensus.cc:348] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d51333c07db4e658383a195687d5040" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37879 } } peers { permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 } } peers { permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 } }
I20260809 22:53:44.837227 6825 raft_consensus.cc:374] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:44.837585 6825 raft_consensus.cc:729] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e0f6b3e81d743aebc8a126063abaa13, State: Initialized, Role: FOLLOWER
I20260809 22:53:44.837263 6826 raft_consensus.cc:348] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d51333c07db4e658383a195687d5040" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37879 } } peers { permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 } } peers { permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 } }
I20260809 22:53:44.838133 6826 raft_consensus.cc:374] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:44.838485 6826 raft_consensus.cc:729] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d51333c07db4e658383a195687d5040, State: Initialized, Role: FOLLOWER
I20260809 22:53:44.838354 6825 consensus_queue.cc:260] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d51333c07db4e658383a195687d5040" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37879 } } peers { permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 } } peers { permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 } }
I20260809 22:53:44.839459 6826 consensus_queue.cc:260] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d51333c07db4e658383a195687d5040" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37879 } } peers { permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 } } peers { permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 } }
I20260809 22:53:44.841876 6723 heartbeater.cc:503] Master 127.5.105.190:41455 was elected leader, sending a full tablet report...
I20260809 22:53:44.842062 6825 ts_tablet_manager.cc:1428] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260809 22:53:44.845361 6636 heartbeater.cc:503] Master 127.5.105.190:41455 was elected leader, sending a full tablet report...
I20260809 22:53:44.846362 6826 ts_tablet_manager.cc:1428] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040: Time spent starting tablet: real 0.012s user 0.004s sys 0.006s
I20260809 22:53:44.849615 6828 tablet_bootstrap.cc:492] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8: No bootstrap required, opened a new log
I20260809 22:53:44.850054 6828 ts_tablet_manager.cc:1397] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.010s
I20260809 22:53:44.852658 6828 raft_consensus.cc:348] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d51333c07db4e658383a195687d5040" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37879 } } peers { permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 } } peers { permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 } }
I20260809 22:53:44.853820 6828 raft_consensus.cc:374] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:44.854072 6828 raft_consensus.cc:729] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d2d72c76f74e32a2d9c918701707a8, State: Initialized, Role: FOLLOWER
I20260809 22:53:44.854729 6828 consensus_queue.cc:260] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d51333c07db4e658383a195687d5040" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37879 } } peers { permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 } } peers { permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 } }
I20260809 22:53:44.856739 6809 heartbeater.cc:503] Master 127.5.105.190:41455 was elected leader, sending a full tablet report...
I20260809 22:53:44.857235 6828 ts_tablet_manager.cc:1428] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260809 22:53:44.954080 6833 raft_consensus.cc:482] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:44.954658 6833 raft_consensus.cc:504] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d51333c07db4e658383a195687d5040" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37879 } } peers { permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 } } peers { permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 } }
I20260809 22:53:44.957420 6833 leader_election.cc:288] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e0f6b3e81d743aebc8a126063abaa13 (127.5.105.130:39805), 49d2d72c76f74e32a2d9c918701707a8 (127.5.105.131:45005)
I20260809 22:53:44.969748 6694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34e2ca4820e456da441c0fc7d6ded1e" candidate_uuid: "4d51333c07db4e658383a195687d5040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e0f6b3e81d743aebc8a126063abaa13" is_pre_election: true
I20260809 22:53:44.970583 6694 raft_consensus.cc:2454] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d51333c07db4e658383a195687d5040 in term 0.
I20260809 22:53:44.971938 6569 leader_election.cc:302] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d51333c07db4e658383a195687d5040, 5e0f6b3e81d743aebc8a126063abaa13; no voters:
I20260809 22:53:44.972168 6778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34e2ca4820e456da441c0fc7d6ded1e" candidate_uuid: "4d51333c07db4e658383a195687d5040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d2d72c76f74e32a2d9c918701707a8" is_pre_election: true
I20260809 22:53:44.972653 6833 raft_consensus.cc:2781] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:53:44.972783 6778 raft_consensus.cc:2454] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d51333c07db4e658383a195687d5040 in term 0.
I20260809 22:53:44.972982 6833 raft_consensus.cc:482] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:53:44.973357 6833 raft_consensus.cc:3037] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:44.978238 6833 raft_consensus.cc:504] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d51333c07db4e658383a195687d5040" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37879 } } peers { permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 } } peers { permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 } }
I20260809 22:53:44.979714 6833 leader_election.cc:288] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [CANDIDATE]: Term 1 election: Requested vote from peers 5e0f6b3e81d743aebc8a126063abaa13 (127.5.105.130:39805), 49d2d72c76f74e32a2d9c918701707a8 (127.5.105.131:45005)
I20260809 22:53:44.980412 6694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34e2ca4820e456da441c0fc7d6ded1e" candidate_uuid: "4d51333c07db4e658383a195687d5040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e0f6b3e81d743aebc8a126063abaa13"
I20260809 22:53:44.980672 6778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34e2ca4820e456da441c0fc7d6ded1e" candidate_uuid: "4d51333c07db4e658383a195687d5040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d2d72c76f74e32a2d9c918701707a8"
I20260809 22:53:44.980952 6694 raft_consensus.cc:3037] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:44.981195 6778 raft_consensus.cc:3037] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:44.987862 6778 raft_consensus.cc:2454] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d51333c07db4e658383a195687d5040 in term 1.
I20260809 22:53:44.987855 6694 raft_consensus.cc:2454] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d51333c07db4e658383a195687d5040 in term 1.
I20260809 22:53:44.989161 6568 leader_election.cc:302] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d2d72c76f74e32a2d9c918701707a8, 4d51333c07db4e658383a195687d5040; no voters:
I20260809 22:53:44.989874 6833 raft_consensus.cc:2781] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:53:44.991499 6833 raft_consensus.cc:686] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [term 1 LEADER]: Becoming Leader. State: Replica: 4d51333c07db4e658383a195687d5040, State: Running, Role: LEADER
I20260809 22:53:44.992129 6833 consensus_queue.cc:237] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d51333c07db4e658383a195687d5040" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37879 } } peers { permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 } } peers { permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 } }
I20260809 22:53:44.999683 6494 catalog_manager.cc:5495] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d51333c07db4e658383a195687d5040 (127.5.105.129). New cstate: current_term: 1 leader_uuid: "4d51333c07db4e658383a195687d5040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d51333c07db4e658383a195687d5040" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 } health_report { overall_health: UNKNOWN } } }
I20260809 22:53:45.388967 6833 consensus_queue.cc:1035] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:53:45.400823 6842 consensus_queue.cc:1035] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:53:46.095467 5542 tablet_replica.cc:330] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040: stopping tablet replica
I20260809 22:53:46.095975 5542 raft_consensus.cc:2229] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:53:46.096637 5542 raft_consensus.cc:2258] T d34e2ca4820e456da441c0fc7d6ded1e P 4d51333c07db4e658383a195687d5040 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:47.306535 6847 raft_consensus.cc:482] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d51333c07db4e658383a195687d5040)
I20260809 22:53:47.306985 6847 raft_consensus.cc:504] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d51333c07db4e658383a195687d5040" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37879 } } peers { permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 } } peers { permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 } }
I20260809 22:53:47.308730 6847 leader_election.cc:288] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d51333c07db4e658383a195687d5040 (127.5.105.129:37879), 49d2d72c76f74e32a2d9c918701707a8 (127.5.105.131:45005)
I20260809 22:53:47.321972 6610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34e2ca4820e456da441c0fc7d6ded1e" candidate_uuid: "5e0f6b3e81d743aebc8a126063abaa13" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d51333c07db4e658383a195687d5040" is_pre_election: true
W20260809 22:53:47.323624 6654 leader_election.cc:341] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d51333c07db4e658383a195687d5040 (127.5.105.129:37879): Illegal state: cannot vote while shut down
I20260809 22:53:47.324254 6778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34e2ca4820e456da441c0fc7d6ded1e" candidate_uuid: "5e0f6b3e81d743aebc8a126063abaa13" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "49d2d72c76f74e32a2d9c918701707a8" is_pre_election: true
I20260809 22:53:47.325595 6654 leader_election.cc:302] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e0f6b3e81d743aebc8a126063abaa13; no voters: 49d2d72c76f74e32a2d9c918701707a8, 4d51333c07db4e658383a195687d5040
I20260809 22:53:47.326445 6847 raft_consensus.cc:2726] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 22:53:47.396431 6852 raft_consensus.cc:482] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d51333c07db4e658383a195687d5040)
I20260809 22:53:47.396994 6852 raft_consensus.cc:504] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d51333c07db4e658383a195687d5040" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37879 } } peers { permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 } } peers { permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 } }
I20260809 22:53:47.399256 6852 leader_election.cc:288] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d51333c07db4e658383a195687d5040 (127.5.105.129:37879), 5e0f6b3e81d743aebc8a126063abaa13 (127.5.105.130:39805)
I20260809 22:53:47.415984 6694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34e2ca4820e456da441c0fc7d6ded1e" candidate_uuid: "49d2d72c76f74e32a2d9c918701707a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e0f6b3e81d743aebc8a126063abaa13" is_pre_election: true
I20260809 22:53:47.416621 6694 raft_consensus.cc:2454] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49d2d72c76f74e32a2d9c918701707a8 in term 1.
I20260809 22:53:47.416760 6610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34e2ca4820e456da441c0fc7d6ded1e" candidate_uuid: "49d2d72c76f74e32a2d9c918701707a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d51333c07db4e658383a195687d5040" is_pre_election: true
I20260809 22:53:47.417764 6742 leader_election.cc:302] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d2d72c76f74e32a2d9c918701707a8, 5e0f6b3e81d743aebc8a126063abaa13; no voters:
I20260809 22:53:47.418532 6852 raft_consensus.cc:2781] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260809 22:53:47.418444 6739 leader_election.cc:341] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d51333c07db4e658383a195687d5040 (127.5.105.129:37879): Illegal state: cannot vote while shut down
I20260809 22:53:47.419016 6852 raft_consensus.cc:482] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d51333c07db4e658383a195687d5040)
I20260809 22:53:47.419468 6852 raft_consensus.cc:3037] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:47.424376 6852 raft_consensus.cc:504] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d51333c07db4e658383a195687d5040" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37879 } } peers { permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 } } peers { permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 } }
I20260809 22:53:47.425951 6852 leader_election.cc:288] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [CANDIDATE]: Term 2 election: Requested vote from peers 4d51333c07db4e658383a195687d5040 (127.5.105.129:37879), 5e0f6b3e81d743aebc8a126063abaa13 (127.5.105.130:39805)
I20260809 22:53:47.426913 6610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34e2ca4820e456da441c0fc7d6ded1e" candidate_uuid: "49d2d72c76f74e32a2d9c918701707a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d51333c07db4e658383a195687d5040"
I20260809 22:53:47.427019 6694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34e2ca4820e456da441c0fc7d6ded1e" candidate_uuid: "49d2d72c76f74e32a2d9c918701707a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e0f6b3e81d743aebc8a126063abaa13"
I20260809 22:53:47.427619 6694 raft_consensus.cc:3037] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 1 FOLLOWER]: Advancing to term 2
W20260809 22:53:47.427944 6739 leader_election.cc:341] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 4d51333c07db4e658383a195687d5040 (127.5.105.129:37879): Illegal state: cannot vote while shut down
I20260809 22:53:47.432389 6694 raft_consensus.cc:2454] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49d2d72c76f74e32a2d9c918701707a8 in term 2.
I20260809 22:53:47.433425 6742 leader_election.cc:302] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49d2d72c76f74e32a2d9c918701707a8, 5e0f6b3e81d743aebc8a126063abaa13; no voters: 4d51333c07db4e658383a195687d5040
I20260809 22:53:47.434056 6852 raft_consensus.cc:2781] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 22:53:47.435169 6852 raft_consensus.cc:686] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 2 LEADER]: Becoming Leader. State: Replica: 49d2d72c76f74e32a2d9c918701707a8, State: Running, Role: LEADER
I20260809 22:53:47.435976 6852 consensus_queue.cc:237] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d51333c07db4e658383a195687d5040" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 37879 } } peers { permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 } } peers { permanent_uuid: "49d2d72c76f74e32a2d9c918701707a8" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45005 } }
I20260809 22:53:47.834774 6694 raft_consensus.cc:1261] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 2 FOLLOWER]: Refusing update from remote peer 49d2d72c76f74e32a2d9c918701707a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 22:53:47.836231 6852 consensus_queue.cc:1035] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e0f6b3e81d743aebc8a126063abaa13" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 39805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 22:53:47.846014 6739 consensus_queue.cc:579] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [LEADER]: The tablet replica hosted on peer 4d51333c07db4e658383a195687d5040 has failed
W20260809 22:53:47.846560 6739 consensus_peers.cc:487] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 -> Peer 4d51333c07db4e658383a195687d5040 (127.5.105.129:37879): Couldn't send request to peer 4d51333c07db4e658383a195687d5040. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260809 22:53:48.131510 5542 tablet_server.cc:178] TabletServer@127.5.105.129:0 shutting down...
I20260809 22:53:48.153891 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:53:48.172043 5542 tablet_server.cc:194] TabletServer@127.5.105.129:0 shutdown complete.
I20260809 22:53:48.180172 5542 tablet_server.cc:178] TabletServer@127.5.105.130:0 shutting down...
I20260809 22:53:48.195765 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:53:48.196326 5542 tablet_replica.cc:330] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13: stopping tablet replica
I20260809 22:53:48.196795 5542 raft_consensus.cc:2229] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 22:53:48.197206 5542 raft_consensus.cc:2258] T d34e2ca4820e456da441c0fc7d6ded1e P 5e0f6b3e81d743aebc8a126063abaa13 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:48.216368 5542 tablet_server.cc:194] TabletServer@127.5.105.130:0 shutdown complete.
I20260809 22:53:48.226920 5542 tablet_server.cc:178] TabletServer@127.5.105.131:0 shutting down...
I20260809 22:53:48.241880 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:53:48.242503 5542 tablet_replica.cc:330] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8: stopping tablet replica
I20260809 22:53:48.242957 5542 raft_consensus.cc:2229] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 2 LEADER]: Raft consensus shutting down.
I20260809 22:53:48.243651 5542 raft_consensus.cc:2258] T d34e2ca4820e456da441c0fc7d6ded1e P 49d2d72c76f74e32a2d9c918701707a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:48.252121 5542 tablet_server.cc:194] TabletServer@127.5.105.131:0 shutdown complete.
I20260809 22:53:48.259467 5542 master.cc:537] Master@127.5.105.190:41455 shutting down...
I20260809 22:53:48.272814 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:53:48.273293 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:48.273578 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P c44a3856917a4a2b80689e94b4994a69: stopping tablet replica
I20260809 22:53:48.290498 5542 master.cc:552] Master@127.5.105.190:41455 shutdown complete.
[ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (4143 ms)
[ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20260809 22:53:48.314041 5542 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.5.105.190:38621
I20260809 22:53:48.315161 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:48.320001 6867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:48.321349 6868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:48.324008 5542 server_base.cc:1034] running on GCE node
W20260809 22:53:48.324154 6870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:48.325043 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:48.325212 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:48.325420 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316028325381 us; error 0 us; skew 500 ppm
I20260809 22:53:48.325986 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:48.333410 5542 webserver.cc:460] Webserver started at http://127.5.105.190:35099/ using document root <none> and password file <none>
I20260809 22:53:48.333928 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:48.334125 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:48.334393 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:48.335537 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786316013627029-5542-0/minicluster-data/master-0-root/instance:
uuid: "b2b255b22a414adca934058545cfef4b"
format_stamp: "Formatted at 2026-08-09 22:53:48 on dist-test-slave-gp6n"
I20260809 22:53:48.340081 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 22:53:48.343405 6877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:48.344205 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 22:53:48.344491 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786316013627029-5542-0/minicluster-data/master-0-root
uuid: "b2b255b22a414adca934058545cfef4b"
format_stamp: "Formatted at 2026-08-09 22:53:48 on dist-test-slave-gp6n"
I20260809 22:53:48.344784 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786316013627029-5542-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786316013627029-5542-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786316013627029-5542-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:48.357191 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:48.358196 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:48.393883 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.190:38621
I20260809 22:53:48.393991 6936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.190:38621 every 8 connection(s)
I20260809 22:53:48.397737 6937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:48.406865 6937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b: Bootstrap starting.
I20260809 22:53:48.410972 6937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:48.414915 6937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b: No bootstrap required, opened a new log
I20260809 22:53:48.416750 6937 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2b255b22a414adca934058545cfef4b" member_type: VOTER }
I20260809 22:53:48.417140 6937 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:48.417388 6937 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b255b22a414adca934058545cfef4b, State: Initialized, Role: FOLLOWER
I20260809 22:53:48.417968 6937 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2b255b22a414adca934058545cfef4b" member_type: VOTER }
I20260809 22:53:48.418422 6937 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:53:48.418649 6937 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:53:48.418895 6937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:48.422969 6937 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2b255b22a414adca934058545cfef4b" member_type: VOTER }
I20260809 22:53:48.423473 6937 leader_election.cc:302] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2b255b22a414adca934058545cfef4b; no voters:
I20260809 22:53:48.424595 6937 leader_election.cc:288] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:53:48.424939 6940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:53:48.426222 6940 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [term 1 LEADER]: Becoming Leader. State: Replica: b2b255b22a414adca934058545cfef4b, State: Running, Role: LEADER
I20260809 22:53:48.426891 6940 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2b255b22a414adca934058545cfef4b" member_type: VOTER }
I20260809 22:53:48.427527 6937 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:53:48.429493 6942 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2b255b22a414adca934058545cfef4b. Latest consensus state: current_term: 1 leader_uuid: "b2b255b22a414adca934058545cfef4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2b255b22a414adca934058545cfef4b" member_type: VOTER } }
I20260809 22:53:48.429483 6941 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2b255b22a414adca934058545cfef4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2b255b22a414adca934058545cfef4b" member_type: VOTER } }
I20260809 22:53:48.430155 6942 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:48.430281 6941 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:48.433061 6945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:53:48.438186 6945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:53:48.441866 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 22:53:48.446591 6945 catalog_manager.cc:1347] Generated new cluster ID: fda9e839aa3b4124a4e9c045f16dd05b
I20260809 22:53:48.446836 6945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:53:48.477645 6945 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:53:48.479295 6945 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:53:48.496346 6945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b: Generated new TSK 0
I20260809 22:53:48.497179 6945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:53:48.508000 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:48.514448 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:48.514832 6962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:48.518397 6964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:48.519281 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:48.520344 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:48.520615 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:48.520825 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316028520811 us; error 0 us; skew 500 ppm
I20260809 22:53:48.521596 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:48.524904 5542 webserver.cc:460] Webserver started at http://127.5.105.129:33075/ using document root <none> and password file <none>
I20260809 22:53:48.525585 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:48.525846 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:48.526206 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:48.527894 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786316013627029-5542-0/minicluster-data/ts-0-root/instance:
uuid: "8513417cf44a4bb9ad5f97294a8ed8d3"
format_stamp: "Formatted at 2026-08-09 22:53:48 on dist-test-slave-gp6n"
I20260809 22:53:48.535564 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260809 22:53:48.540462 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:48.541365 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 22:53:48.541776 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "8513417cf44a4bb9ad5f97294a8ed8d3"
format_stamp: "Formatted at 2026-08-09 22:53:48 on dist-test-slave-gp6n"
I20260809 22:53:48.542121 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:48.564906 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:48.566004 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:48.567628 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:53:48.569957 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:53:48.570168 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:48.570459 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:53:48.570638 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:48.616343 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:34171
I20260809 22:53:48.616459 7036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:34171 every 8 connection(s)
I20260809 22:53:48.631848 7037 heartbeater.cc:347] Connected to a master server at 127.5.105.190:38621
I20260809 22:53:48.632226 7037 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:48.632872 7037 heartbeater.cc:511] Master 127.5.105.190:38621 requested a full tablet report, sending...
I20260809 22:53:48.635084 6897 ts_manager.cc:194] Registered new tserver with Master: 8513417cf44a4bb9ad5f97294a8ed8d3 (127.5.105.129:34171)
I20260809 22:53:48.635201 5542 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.01564563s
I20260809 22:53:48.637429 6897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51356
I20260809 22:53:48.657361 5542 tablet_server.cc:178] TabletServer@127.5.105.129:0 shutting down...
I20260809 22:53:48.674057 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:53:48.689644 5542 tablet_server.cc:194] TabletServer@127.5.105.129:0 shutdown complete.
I20260809 22:53:48.696975 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 22:53:48.699541 5542 tablet_server.cc:93] deduplicated master addresses: 127.5.105.190:38621
W20260809 22:53:48.702958 7047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:48.703824 7049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:48.705102 7052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:48.705185 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:48.706193 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:48.706362 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:48.706597 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316028706557 us; error 0 us; skew 500 ppm
I20260809 22:53:48.707183 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:48.709424 5542 webserver.cc:460] Webserver started at http://127.5.105.129:33075/ using document root <none> and password file <none>
I20260809 22:53:48.709914 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:48.710155 5542 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260809 22:53:48.714018 5542 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:53:48.716534 7057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:48.717248 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 22:53:48.717475 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "8513417cf44a4bb9ad5f97294a8ed8d3"
format_stamp: "Formatted at 2026-08-09 22:53:48 on dist-test-slave-gp6n"
I20260809 22:53:48.717721 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:48.733105 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:48.734062 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:48.735563 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:53:48.737775 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:53:48.737950 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:48.738179 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:53:48.738314 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:48.778179 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:34171
I20260809 22:53:48.778261 7126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:34171 every 8 connection(s)
I20260809 22:53:48.784873 5542 tablet_server.cc:178] TabletServer@127.5.105.129:34171 shutting down...
I20260809 22:53:48.804692 7127 heartbeater.cc:347] Connected to a master server at 127.5.105.190:38621
I20260809 22:53:48.805100 7127 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:48.805874 7127 heartbeater.cc:511] Master 127.5.105.190:38621 requested a full tablet report, sending...
I20260809 22:53:48.809866 6897 ts_manager.cc:194] Re-registered known tserver with Master: 8513417cf44a4bb9ad5f97294a8ed8d3 (127.5.105.129:34171)
I20260809 22:53:48.812288 6897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51368
I20260809 22:53:48.815227 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:53:48.833056 5542 tablet_server.cc:194] TabletServer@127.5.105.129:34171 shutdown complete.
I20260809 22:53:48.841786 5542 master.cc:537] Master@127.5.105.190:38621 shutting down...
I20260809 22:53:48.857947 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:53:48.858521 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:48.859071 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P b2b255b22a414adca934058545cfef4b: stopping tablet replica
I20260809 22:53:48.877197 5542 master.cc:552] Master@127.5.105.190:38621 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (583 ms)
[ RUN ] TsTabletManagerITest.TestTableStats
I20260809 22:53:48.897730 5542 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.5.105.190:34015,127.5.105.189:33689,127.5.105.188:45511
I20260809 22:53:48.899012 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:48.903616 7135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:48.904963 7136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:48.906869 7138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:48.907117 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:48.908128 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:48.908304 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:48.908417 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316028908409 us; error 0 us; skew 500 ppm
I20260809 22:53:48.908850 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:48.911170 5542 webserver.cc:460] Webserver started at http://127.5.105.190:46497/ using document root <none> and password file <none>
I20260809 22:53:48.911641 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:48.911825 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:48.912061 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:48.913054 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-0-root/instance:
uuid: "23b696fe94a441ba93177a12ff465c26"
format_stamp: "Formatted at 2026-08-09 22:53:48 on dist-test-slave-gp6n"
I20260809 22:53:48.917459 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 22:53:48.920739 7143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:48.921537 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 22:53:48.921808 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-0-root
uuid: "23b696fe94a441ba93177a12ff465c26"
format_stamp: "Formatted at 2026-08-09 22:53:48 on dist-test-slave-gp6n"
I20260809 22:53:48.922113 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:48.945092 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:48.946110 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:48.981519 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.190:34015
I20260809 22:53:48.981616 7199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.190:34015 every 8 connection(s)
I20260809 22:53:48.985172 7200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:48.985410 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:48.990502 7202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:48.991608 7204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:48.993327 7200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } has no permanent_uuid. Determining permanent_uuid...
W20260809 22:53:48.994789 7206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:48.997272 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:48.998719 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:48.998976 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:48.999171 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316028999154 us; error 0 us; skew 500 ppm
I20260809 22:53:48.999716 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:49.002903 5542 webserver.cc:460] Webserver started at http://127.5.105.189:36489/ using document root <none> and password file <none>
I20260809 22:53:49.003364 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:49.003592 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:49.003895 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:49.005323 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-1-root/instance:
uuid: "75d10a05561a482f9823bab385db5ed3"
format_stamp: "Formatted at 2026-08-09 22:53:49 on dist-test-slave-gp6n"
I20260809 22:53:49.009752 7200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:53:49.010640 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
W20260809 22:53:49.012260 7145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.105.189:33689: connect: Connection refused (error 111)
I20260809 22:53:49.014460 7216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:49.015300 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:53:49.015650 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-1-root
uuid: "75d10a05561a482f9823bab385db5ed3"
format_stamp: "Formatted at 2026-08-09 22:53:49 on dist-test-slave-gp6n"
W20260809 22:53:49.015836 7200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.105.189:33689: Network error: Client connection negotiation failed: client connection to 127.5.105.189:33689: connect: Connection refused (error 111)
I20260809 22:53:49.016029 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:49.035858 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:49.036854 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:49.054381 7200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } attempt: 1
W20260809 22:53:49.058430 7200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.105.189:33689: Network error: Client connection negotiation failed: client connection to 127.5.105.189:33689: connect: Connection refused (error 111)
I20260809 22:53:49.074581 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.189:33689
I20260809 22:53:49.074673 7275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.189:33689 every 8 connection(s)
I20260809 22:53:49.078496 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 22:53:49.078560 7277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 22:53:49.085306 7279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:49.087523 7277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } has no permanent_uuid. Determining permanent_uuid...
W20260809 22:53:49.091174 7282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:49.089918 7280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:49.094105 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:49.095245 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:49.095496 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:49.095716 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316029095699 us; error 0 us; skew 500 ppm
I20260809 22:53:49.096293 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:49.098819 7200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } attempt: 2
I20260809 22:53:49.099362 5542 webserver.cc:460] Webserver started at http://127.5.105.188:44935/ using document root <none> and password file <none>
I20260809 22:53:49.100044 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:49.100315 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:49.100660 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:49.102267 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-2-root/instance:
uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac"
format_stamp: "Formatted at 2026-08-09 22:53:49 on dist-test-slave-gp6n"
I20260809 22:53:49.106938 7277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:53:49.108989 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 22:53:49.113513 7290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:49.114738 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 22:53:49.115089 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-2-root
uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac"
format_stamp: "Formatted at 2026-08-09 22:53:49 on dist-test-slave-gp6n"
I20260809 22:53:49.115477 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:49.116050 7200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } has no permanent_uuid. Determining permanent_uuid...
W20260809 22:53:49.120671 7200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.105.188:45511: Network error: Client connection negotiation failed: client connection to 127.5.105.188:45511: connect: Connection refused (error 111)
I20260809 22:53:49.125380 7277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } has no permanent_uuid. Determining permanent_uuid...
W20260809 22:53:49.130174 7277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.105.188:45511: Network error: Client connection negotiation failed: client connection to 127.5.105.188:45511: connect: Connection refused (error 111)
I20260809 22:53:49.149212 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:49.150322 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:49.156107 7200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } attempt: 1
W20260809 22:53:49.160406 7200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.105.188:45511: Network error: Client connection negotiation failed: client connection to 127.5.105.188:45511: connect: Connection refused (error 111)
I20260809 22:53:49.163684 7277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } attempt: 1
W20260809 22:53:49.168414 7277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.105.188:45511: Network error: Client connection negotiation failed: client connection to 127.5.105.188:45511: connect: Connection refused (error 111)
I20260809 22:53:49.191001 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.188:45511
I20260809 22:53:49.191089 7350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.188:45511 every 8 connection(s)
I20260809 22:53:49.194348 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 22:53:49.195173 7351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:49.200664 7351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:53:49.206943 7277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } attempt: 2
I20260809 22:53:49.214272 7351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:53:49.226823 7277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3: Bootstrap starting.
I20260809 22:53:49.231137 7277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:49.231921 7351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:53:49.235332 7277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3: No bootstrap required, opened a new log
I20260809 22:53:49.235884 7200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } attempt: 2
I20260809 22:53:49.237527 7277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:49.238297 7277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:49.238612 7277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d10a05561a482f9823bab385db5ed3, State: Initialized, Role: FOLLOWER
I20260809 22:53:49.239339 7277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:49.241753 7359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:49.242563 7359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:49.243086 7277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:53:49.255157 7351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac: Bootstrap starting.
I20260809 22:53:49.259596 7200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: Bootstrap starting.
I20260809 22:53:49.260958 7351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac: Neither blocks nor log segments found. Creating new log.
W20260809 22:53:49.261992 7372 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 22:53:49.262290 7372 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 22:53:49.265230 7200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:49.265630 7351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac: No bootstrap required, opened a new log
I20260809 22:53:49.268181 7351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:49.268893 7351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:49.269188 7351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed55e55e8fd4ad4830e8fd57d1d7cac, State: Initialized, Role: FOLLOWER
I20260809 22:53:49.269915 7200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: No bootstrap required, opened a new log
I20260809 22:53:49.270058 7351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:49.272027 7376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:49.272629 7376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:49.272579 7200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:49.273425 7200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:49.273651 7351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:53:49.273710 7200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b696fe94a441ba93177a12ff465c26, State: Initialized, Role: FOLLOWER
I20260809 22:53:49.274489 7200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:49.276397 7377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:49.277096 7377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:49.278481 7200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:53:49.288854 7359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:49.289418 7359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
W20260809 22:53:49.291397 7394 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 22:53:49.291800 7394 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 22:53:49.292344 7359 leader_election.cc:288] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015), 1ed55e55e8fd4ad4830e8fd57d1d7cac (127.5.105.188:45511)
I20260809 22:53:49.292796 7401 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 22:53:49.293239 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20260809 22:53:49.293052 7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75d10a05561a482f9823bab385db5ed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b696fe94a441ba93177a12ff465c26" is_pre_election: true
I20260809 22:53:49.293057 7323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75d10a05561a482f9823bab385db5ed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" is_pre_election: true
I20260809 22:53:49.293620 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20260809 22:53:49.293552 7376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:49.293802 7173 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75d10a05561a482f9823bab385db5ed3 in term 0.
I20260809 22:53:49.294129 7376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:49.295123 7323 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75d10a05561a482f9823bab385db5ed3 in term 0.
I20260809 22:53:49.295073 7219 leader_election.cc:302] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b696fe94a441ba93177a12ff465c26, 75d10a05561a482f9823bab385db5ed3; no voters:
I20260809 22:53:49.295625 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 22:53:49.296226 7359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:53:49.296413 7376 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015), 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689)
I20260809 22:53:49.296685 7359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:53:49.297147 7359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:49.299292 7249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75d10a05561a482f9823bab385db5ed3" is_pre_election: true
I20260809 22:53:49.300689 7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b696fe94a441ba93177a12ff465c26" is_pre_election: true
I20260809 22:53:49.301160 7173 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ed55e55e8fd4ad4830e8fd57d1d7cac in term 0.
I20260809 22:53:49.302361 7293 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac, 23b696fe94a441ba93177a12ff465c26; no voters:
I20260809 22:53:49.303248 7376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:53:49.303645 7376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:53:49.304031 7376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 0 FOLLOWER]: Advancing to term 1
W20260809 22:53:49.304278 7402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:49.305313 7359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:49.306877 7249 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ed55e55e8fd4ad4830e8fd57d1d7cac in current term 1: Already voted for candidate 75d10a05561a482f9823bab385db5ed3 in this term.
W20260809 22:53:49.307039 7403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:49.311618 7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75d10a05561a482f9823bab385db5ed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b696fe94a441ba93177a12ff465c26"
I20260809 22:53:49.312181 7173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:49.312300 7376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:49.313138 7323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75d10a05561a482f9823bab385db5ed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac"
I20260809 22:53:49.314133 7323 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 75d10a05561a482f9823bab385db5ed3 in current term 1: Already voted for candidate 1ed55e55e8fd4ad4830e8fd57d1d7cac in this term.
I20260809 22:53:49.314484 7376 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 1 election: Requested vote from peers 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015), 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689)
I20260809 22:53:49.316684 7249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75d10a05561a482f9823bab385db5ed3"
I20260809 22:53:49.317499 7173 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75d10a05561a482f9823bab385db5ed3 in term 1.
I20260809 22:53:49.317692 7249 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ed55e55e8fd4ad4830e8fd57d1d7cac in current term 1: Already voted for candidate 75d10a05561a482f9823bab385db5ed3 in this term.
I20260809 22:53:49.319015 7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b696fe94a441ba93177a12ff465c26"
I20260809 22:53:49.318817 7219 leader_election.cc:302] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23b696fe94a441ba93177a12ff465c26, 75d10a05561a482f9823bab385db5ed3; no voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac
I20260809 22:53:49.319492 7173 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ed55e55e8fd4ad4830e8fd57d1d7cac in current term 1: Already voted for candidate 75d10a05561a482f9823bab385db5ed3 in this term.
I20260809 22:53:49.321367 7292 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac; no voters: 23b696fe94a441ba93177a12ff465c26, 75d10a05561a482f9823bab385db5ed3
I20260809 22:53:49.322569 7376 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 22:53:49.323712 7359 leader_election.cc:288] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [CANDIDATE]: Term 1 election: Requested vote from peers 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015), 1ed55e55e8fd4ad4830e8fd57d1d7cac (127.5.105.188:45511)
I20260809 22:53:49.324466 7359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:53:49.325055 7359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 1 LEADER]: Becoming Leader. State: Replica: 75d10a05561a482f9823bab385db5ed3, State: Running, Role: LEADER
I20260809 22:53:49.326153 7359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:49.333328 7407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75d10a05561a482f9823bab385db5ed3. Latest consensus state: current_term: 1 leader_uuid: "75d10a05561a482f9823bab385db5ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:49.334193 7407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:49.335321 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:49.335526 7410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260809 22:53:49.336608 7405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:49.337622 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:49.337882 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:49.338074 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316029338059 us; error 0 us; skew 500 ppm
I20260809 22:53:49.338838 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:49.341765 5542 webserver.cc:460] Webserver started at http://127.5.105.129:44349/ using document root <none> and password file <none>
I20260809 22:53:49.341826 7410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:53:49.342473 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:49.342732 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:49.343057 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:49.344471 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-0-root/instance:
uuid: "493890a07f5f4d34b9f48a3da5421253"
format_stamp: "Formatted at 2026-08-09 22:53:49 on dist-test-slave-gp6n"
I20260809 22:53:49.350584 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 22:53:49.350620 7173 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 1 FOLLOWER]: Refusing update from remote peer 75d10a05561a482f9823bab385db5ed3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:53:49.350622 7323 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 1 FOLLOWER]: Refusing update from remote peer 75d10a05561a482f9823bab385db5ed3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:53:49.352231 7359 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:53:49.352963 7407 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:53:49.355123 7416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:49.361109 5542 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260809 22:53:49.361608 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "493890a07f5f4d34b9f48a3da5421253"
format_stamp: "Formatted at 2026-08-09 22:53:49 on dist-test-slave-gp6n"
I20260809 22:53:49.361132 7377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75d10a05561a482f9823bab385db5ed3. Latest consensus state: current_term: 1 leader_uuid: "75d10a05561a482f9823bab385db5ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:49.362227 7377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:49.362028 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:49.363512 7376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75d10a05561a482f9823bab385db5ed3. Latest consensus state: current_term: 1 leader_uuid: "75d10a05561a482f9823bab385db5ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:49.364257 7376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:49.365828 7413 mvcc.cc:205] Tried to move back new op lower bound from 7316750456207753216 to 7316750456136060928. Current Snapshot: MvccSnapshot[applied={T|T < 7316750456207753216}]
I20260809 22:53:49.371450 7359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75d10a05561a482f9823bab385db5ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:49.372189 7359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:49.373371 7410 catalog_manager.cc:1347] Generated new cluster ID: f511035ecb4f4aaa9b8234496968ea8e
I20260809 22:53:49.372979 7359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75d10a05561a482f9823bab385db5ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:49.373641 7410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:53:49.373677 7359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:49.374094 7376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75d10a05561a482f9823bab385db5ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:49.375002 7376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:49.380270 7377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75d10a05561a482f9823bab385db5ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:49.381021 7377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:49.417969 7410 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:53:49.420164 7410 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:53:49.422349 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:49.423492 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:49.425020 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:53:49.427627 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:53:49.427830 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:49.428088 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:53:49.428241 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:49.444305 7410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3: Generated new TSK 0
I20260809 22:53:49.445068 7410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:53:49.480020 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:33533
I20260809 22:53:49.480417 7496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:33533 every 8 connection(s)
I20260809 22:53:49.491569 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 22:53:49.503841 7497 heartbeater.cc:347] Connected to a master server at 127.5.105.188:45511
I20260809 22:53:49.504247 7497 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:49.506016 7497 heartbeater.cc:511] Master 127.5.105.188:45511 requested a full tablet report, sending...
I20260809 22:53:49.510846 7312 ts_manager.cc:194] Registered new tserver with Master: 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533)
W20260809 22:53:49.511113 7505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:49.511634 7504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:49.519410 7499 heartbeater.cc:347] Connected to a master server at 127.5.105.190:34015
I20260809 22:53:49.519835 7499 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:49.520804 7499 heartbeater.cc:511] Master 127.5.105.190:34015 requested a full tablet report, sending...
I20260809 22:53:49.524287 7161 ts_manager.cc:194] Registered new tserver with Master: 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533)
W20260809 22:53:49.525429 7507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:49.526018 7498 heartbeater.cc:347] Connected to a master server at 127.5.105.189:33689
I20260809 22:53:49.526435 7498 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:49.527295 7498 heartbeater.cc:511] Master 127.5.105.189:33689 requested a full tablet report, sending...
I20260809 22:53:49.528724 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:49.529821 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260809 22:53:49.529920 7238 ts_manager.cc:194] Registered new tserver with Master: 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533)
W20260809 22:53:49.530076 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:49.530373 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316029530356 us; error 0 us; skew 500 ppm
I20260809 22:53:49.531083 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:49.531972 7238 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50900
I20260809 22:53:49.535180 5542 webserver.cc:460] Webserver started at http://127.5.105.130:44243/ using document root <none> and password file <none>
I20260809 22:53:49.535740 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:49.535961 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:49.536273 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:49.537400 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-1-root/instance:
uuid: "055f6d77fb794a629017dd736c1a6060"
format_stamp: "Formatted at 2026-08-09 22:53:49 on dist-test-slave-gp6n"
I20260809 22:53:49.541956 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260809 22:53:49.545331 7512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:49.546082 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 22:53:49.546382 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-1-root
uuid: "055f6d77fb794a629017dd736c1a6060"
format_stamp: "Formatted at 2026-08-09 22:53:49 on dist-test-slave-gp6n"
I20260809 22:53:49.546728 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:49.559609 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:49.560976 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:49.563251 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:53:49.566663 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:53:49.566998 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:49.567354 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:53:49.567619 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:49.635567 7498 heartbeater.cc:503] Master 127.5.105.189:33689 was elected leader, sending a full tablet report...
I20260809 22:53:49.655625 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.130:33821
I20260809 22:53:49.655711 7582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.130:33821 every 8 connection(s)
I20260809 22:53:49.677115 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 22:53:49.683657 7584 heartbeater.cc:347] Connected to a master server at 127.5.105.188:45511
I20260809 22:53:49.684131 7584 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:49.685274 7584 heartbeater.cc:511] Master 127.5.105.188:45511 requested a full tablet report, sending...
I20260809 22:53:49.689066 7312 ts_manager.cc:194] Registered new tserver with Master: 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
W20260809 22:53:49.691921 7593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:49.700700 7586 heartbeater.cc:347] Connected to a master server at 127.5.105.190:34015
I20260809 22:53:49.701251 7586 heartbeater.cc:464] Registering TS with master...
W20260809 22:53:49.702013 7592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:49.702929 7586 heartbeater.cc:511] Master 127.5.105.190:34015 requested a full tablet report, sending...
I20260809 22:53:49.706620 7161 ts_manager.cc:194] Registered new tserver with Master: 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:49.708244 7585 heartbeater.cc:347] Connected to a master server at 127.5.105.189:33689
I20260809 22:53:49.708611 7585 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:49.709333 7585 heartbeater.cc:511] Master 127.5.105.189:33689 requested a full tablet report, sending...
I20260809 22:53:49.709388 5542 server_base.cc:1034] running on GCE node
W20260809 22:53:49.711134 7595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:49.711472 7238 ts_manager.cc:194] Registered new tserver with Master: 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:49.712064 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:49.712332 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:49.712550 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316029712532 us; error 0 us; skew 500 ppm
I20260809 22:53:49.713266 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:49.713617 7238 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50910
I20260809 22:53:49.716238 5542 webserver.cc:460] Webserver started at http://127.5.105.131:40671/ using document root <none> and password file <none>
I20260809 22:53:49.716886 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:49.717133 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:53:49.717394 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:53:49.718569 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-2-root/instance:
uuid: "d8db860deafb41209b9e337a987f2887"
format_stamp: "Formatted at 2026-08-09 22:53:49 on dist-test-slave-gp6n"
I20260809 22:53:49.723094 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260809 22:53:49.726248 7601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:49.726982 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 22:53:49.727227 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-2-root
uuid: "d8db860deafb41209b9e337a987f2887"
format_stamp: "Formatted at 2026-08-09 22:53:49 on dist-test-slave-gp6n"
I20260809 22:53:49.727463 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:49.764885 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:49.765944 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:49.767128 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:53:49.769233 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:53:49.769451 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:49.769704 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:53:49.769858 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:49.817251 7585 heartbeater.cc:503] Master 127.5.105.189:33689 was elected leader, sending a full tablet report...
I20260809 22:53:49.848660 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.131:33583
I20260809 22:53:49.848767 7671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.131:33583 every 8 connection(s)
I20260809 22:53:49.874136 7673 heartbeater.cc:347] Connected to a master server at 127.5.105.188:45511
I20260809 22:53:49.874612 7673 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:49.875442 7673 heartbeater.cc:511] Master 127.5.105.188:45511 requested a full tablet report, sending...
I20260809 22:53:49.878181 7675 heartbeater.cc:347] Connected to a master server at 127.5.105.190:34015
I20260809 22:53:49.878316 7312 ts_manager.cc:194] Registered new tserver with Master: d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:49.878542 7675 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:49.879429 7675 heartbeater.cc:511] Master 127.5.105.190:34015 requested a full tablet report, sending...
I20260809 22:53:49.882151 7161 ts_manager.cc:194] Registered new tserver with Master: d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:49.883534 7674 heartbeater.cc:347] Connected to a master server at 127.5.105.189:33689
I20260809 22:53:49.883899 7674 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:49.884704 7674 heartbeater.cc:511] Master 127.5.105.189:33689 requested a full tablet report, sending...
I20260809 22:53:49.886852 7238 ts_manager.cc:194] Registered new tserver with Master: d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:49.886998 5542 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.027689801s
I20260809 22:53:49.889376 7238 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50914
I20260809 22:53:49.933948 7238 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20260809 22:53:49.935720 7238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:53:49.993301 7674 heartbeater.cc:503] Master 127.5.105.189:33689 was elected leader, sending a full tablet report...
I20260809 22:53:50.025949 7542 tablet_service.cc:1467] Processing CreateTablet for tablet 19ef6c21cc3c497483e1ddb9b9fe17ef (DEFAULT_TABLE table=test-table [id=842a76a4c8694fd187d3c04383286ae1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:50.026566 7633 tablet_service.cc:1467] Processing CreateTablet for tablet 19ef6c21cc3c497483e1ddb9b9fe17ef (DEFAULT_TABLE table=test-table [id=842a76a4c8694fd187d3c04383286ae1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:50.027429 7541 tablet_service.cc:1467] Processing CreateTablet for tablet 119c8382d78c4413aefb2d2d9077d47e (DEFAULT_TABLE table=test-table [id=842a76a4c8694fd187d3c04383286ae1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:50.027467 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ef6c21cc3c497483e1ddb9b9fe17ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:50.028198 7633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ef6c21cc3c497483e1ddb9b9fe17ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:50.028789 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119c8382d78c4413aefb2d2d9077d47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:50.029647 7631 tablet_service.cc:1467] Processing CreateTablet for tablet 119c8382d78c4413aefb2d2d9077d47e (DEFAULT_TABLE table=test-table [id=842a76a4c8694fd187d3c04383286ae1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:50.029989 7540 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad6955771cc4caa8eda877ded5a808a (DEFAULT_TABLE table=test-table [id=842a76a4c8694fd187d3c04383286ae1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:50.031418 7630 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad6955771cc4caa8eda877ded5a808a (DEFAULT_TABLE table=test-table [id=842a76a4c8694fd187d3c04383286ae1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:50.034181 7631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119c8382d78c4413aefb2d2d9077d47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:50.039055 7456 tablet_service.cc:1467] Processing CreateTablet for tablet 19ef6c21cc3c497483e1ddb9b9fe17ef (DEFAULT_TABLE table=test-table [id=842a76a4c8694fd187d3c04383286ae1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:50.040741 7454 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad6955771cc4caa8eda877ded5a808a (DEFAULT_TABLE table=test-table [id=842a76a4c8694fd187d3c04383286ae1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:50.041683 7630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad6955771cc4caa8eda877ded5a808a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:50.044595 7454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad6955771cc4caa8eda877ded5a808a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:50.045795 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad6955771cc4caa8eda877ded5a808a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:50.047557 7456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ef6c21cc3c497483e1ddb9b9fe17ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:50.048516 7455 tablet_service.cc:1467] Processing CreateTablet for tablet 119c8382d78c4413aefb2d2d9077d47e (DEFAULT_TABLE table=test-table [id=842a76a4c8694fd187d3c04383286ae1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260809 22:53:50.054666 7455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119c8382d78c4413aefb2d2d9077d47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:53:50.092746 7706 tablet_bootstrap.cc:492] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887: Bootstrap starting.
I20260809 22:53:50.102566 7707 tablet_bootstrap.cc:492] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060: Bootstrap starting.
I20260809 22:53:50.116839 7707 tablet_bootstrap.cc:654] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:50.116173 7708 tablet_bootstrap.cc:492] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253: Bootstrap starting.
I20260809 22:53:50.132849 7706 tablet_bootstrap.cc:654] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:50.139307 7707 tablet_bootstrap.cc:492] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060: No bootstrap required, opened a new log
I20260809 22:53:50.140492 7707 ts_tablet_manager.cc:1397] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.013s
I20260809 22:53:50.142305 7708 tablet_bootstrap.cc:654] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:50.147307 7707 raft_consensus.cc:348] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.148855 7707 raft_consensus.cc:374] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:50.149387 7707 raft_consensus.cc:729] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055f6d77fb794a629017dd736c1a6060, State: Initialized, Role: FOLLOWER
I20260809 22:53:50.151006 7707 consensus_queue.cc:260] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.167409 7707 ts_tablet_manager.cc:1428] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060: Time spent starting tablet: real 0.026s user 0.009s sys 0.009s
I20260809 22:53:50.169394 7707 tablet_bootstrap.cc:492] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060: Bootstrap starting.
I20260809 22:53:50.169605 7706 tablet_bootstrap.cc:492] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887: No bootstrap required, opened a new log
I20260809 22:53:50.170372 7706 ts_tablet_manager.cc:1397] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.024s
I20260809 22:53:50.197124 7706 raft_consensus.cc:348] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:50.200407 7706 raft_consensus.cc:374] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:50.201021 7706 raft_consensus.cc:729] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8db860deafb41209b9e337a987f2887, State: Initialized, Role: FOLLOWER
I20260809 22:53:50.205590 7707 tablet_bootstrap.cc:654] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:50.206092 7706 consensus_queue.cc:260] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:50.207625 7708 tablet_bootstrap.cc:492] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253: No bootstrap required, opened a new log
I20260809 22:53:50.208395 7708 ts_tablet_manager.cc:1397] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253: Time spent bootstrapping tablet: real 0.093s user 0.027s sys 0.010s
I20260809 22:53:50.214313 7708 raft_consensus.cc:348] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:50.219931 7708 raft_consensus.cc:374] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:50.220443 7708 raft_consensus.cc:729] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493890a07f5f4d34b9f48a3da5421253, State: Initialized, Role: FOLLOWER
I20260809 22:53:50.221689 7708 consensus_queue.cc:260] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:50.229923 7711 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:50.235323 7706 ts_tablet_manager.cc:1428] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887: Time spent starting tablet: real 0.064s user 0.020s sys 0.016s
I20260809 22:53:50.236974 7706 tablet_bootstrap.cc:492] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887: Bootstrap starting.
I20260809 22:53:50.247011 7711 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.251657 7708 ts_tablet_manager.cc:1428] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253: Time spent starting tablet: real 0.043s user 0.016s sys 0.019s
I20260809 22:53:50.254724 7711 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:50.269090 7714 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:50.270020 7714 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:50.282511 7716 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:50.283950 7716 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:50.322285 7707 tablet_bootstrap.cc:492] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060: No bootstrap required, opened a new log
I20260809 22:53:50.325264 7707 ts_tablet_manager.cc:1397] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060: Time spent bootstrapping tablet: real 0.156s user 0.021s sys 0.095s
I20260809 22:53:50.327975 7714 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:50.333328 7394 catalog_manager.cc:1259] Loaded cluster ID: f511035ecb4f4aaa9b8234496968ea8e
I20260809 22:53:50.333784 7394 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac: loading cluster ID for follower catalog manager: success
I20260809 22:53:50.331698 7706 tablet_bootstrap.cc:654] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:50.354635 7708 tablet_bootstrap.cc:492] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253: Bootstrap starting.
I20260809 22:53:50.361241 7707 raft_consensus.cc:348] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:50.363595 7707 raft_consensus.cc:374] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:50.364403 7707 raft_consensus.cc:729] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055f6d77fb794a629017dd736c1a6060, State: Initialized, Role: FOLLOWER
I20260809 22:53:50.365828 7707 consensus_queue.cc:260] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:50.367393 7401 catalog_manager.cc:1259] Loaded cluster ID: f511035ecb4f4aaa9b8234496968ea8e
I20260809 22:53:50.366290 7394 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac: acquiring CA information for follower catalog manager: success
I20260809 22:53:50.368435 7401 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: loading cluster ID for follower catalog manager: success
I20260809 22:53:50.385640 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:50.387537 7554 raft_consensus.cc:2454] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8db860deafb41209b9e337a987f2887 in term 0.
I20260809 22:53:50.384608 7716 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:50.395572 7603 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters:
I20260809 22:53:50.397082 7714 raft_consensus.cc:2781] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:53:50.397672 7714 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:53:50.398221 7714 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:50.398260 7707 ts_tablet_manager.cc:1428] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060: Time spent starting tablet: real 0.072s user 0.011s sys 0.000s
I20260809 22:53:50.403609 7394 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 22:53:50.471297 7714 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:50.478966 7707 tablet_bootstrap.cc:492] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060: Bootstrap starting.
I20260809 22:53:50.508773 7714 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 1 election: Requested vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:50.533744 7401 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: acquiring CA information for follower catalog manager: success
I20260809 22:53:50.543414 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060"
I20260809 22:53:50.557165 7554 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Advancing to term 1
W20260809 22:53:50.562021 7513 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.5.105.129:33533, user_credentials={real_user=slave}} blocked reactor thread for 73244.3us
W20260809 22:53:50.564524 7602 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.5.105.129:33533, user_credentials={real_user=slave}} blocked reactor thread for 45092.1us
I20260809 22:53:50.566267 7707 tablet_bootstrap.cc:654] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:50.564100 7708 tablet_bootstrap.cc:654] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:50.572593 7554 raft_consensus.cc:2454] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8db860deafb41209b9e337a987f2887 in term 1.
I20260809 22:53:50.574515 7603 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters:
I20260809 22:53:50.575438 7714 raft_consensus.cc:2781] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:53:50.580224 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:50.584322 7554 raft_consensus.cc:2379] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 493890a07f5f4d34b9f48a3da5421253 in current term 1: Already voted for candidate d8db860deafb41209b9e337a987f2887 in this term.
I20260809 22:53:50.584051 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
W20260809 22:53:50.587047 7513 leader_election.cc:341] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533): Illegal state: must be running to vote when last-logged opid is not known
I20260809 22:53:50.589986 7706 tablet_bootstrap.cc:492] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887: No bootstrap required, opened a new log
I20260809 22:53:50.590915 7706 ts_tablet_manager.cc:1397] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887: Time spent bootstrapping tablet: real 0.354s user 0.010s sys 0.038s
I20260809 22:53:50.593430 7714 raft_consensus.cc:686] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 1 LEADER]: Becoming Leader. State: Replica: d8db860deafb41209b9e337a987f2887, State: Running, Role: LEADER
I20260809 22:53:50.600391 7707 tablet_bootstrap.cc:492] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060: No bootstrap required, opened a new log
I20260809 22:53:50.605535 7401 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 22:53:50.637135 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:50.639900 7708 tablet_bootstrap.cc:492] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253: No bootstrap required, opened a new log
I20260809 22:53:50.640008 7707 ts_tablet_manager.cc:1397] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060: Time spent bootstrapping tablet: real 0.240s user 0.010s sys 0.019s
I20260809 22:53:50.637136 7714 consensus_queue.cc:237] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:50.640518 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:50.643695 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253"
W20260809 22:53:50.643026 7513 leader_election.cc:341] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583): Illegal state: must be running to vote when last-logged opid is not known
I20260809 22:53:50.643952 7706 raft_consensus.cc:348] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.644587 7467 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:50.645002 7513 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060; no voters: 493890a07f5f4d34b9f48a3da5421253, d8db860deafb41209b9e337a987f2887
I20260809 22:53:50.640849 7708 ts_tablet_manager.cc:1397] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253: Time spent bootstrapping tablet: real 0.289s user 0.012s sys 0.014s
I20260809 22:53:50.644991 7706 raft_consensus.cc:374] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:50.647753 7711 raft_consensus.cc:2726] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 22:53:50.648608 7706 raft_consensus.cc:729] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8db860deafb41209b9e337a987f2887, State: Initialized, Role: FOLLOWER
I20260809 22:53:50.647835 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:50.652248 7422 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493890a07f5f4d34b9f48a3da5421253; no voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887
I20260809 22:53:50.650461 7706 consensus_queue.cc:260] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.667168 7706 ts_tablet_manager.cc:1428] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887: Time spent starting tablet: real 0.075s user 0.014s sys 0.031s
I20260809 22:53:50.669529 7706 tablet_bootstrap.cc:492] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887: Bootstrap starting.
I20260809 22:53:50.669612 7467 raft_consensus.cc:2454] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8db860deafb41209b9e337a987f2887 in term 1.
I20260809 22:53:50.672237 7707 raft_consensus.cc:348] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.675264 7707 raft_consensus.cc:374] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:50.675733 7707 raft_consensus.cc:729] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055f6d77fb794a629017dd736c1a6060, State: Initialized, Role: FOLLOWER
I20260809 22:53:50.676680 7707 consensus_queue.cc:260] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.672730 7708 raft_consensus.cc:348] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.675164 7716 raft_consensus.cc:2726] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 22:53:50.678319 7708 raft_consensus.cc:374] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:50.678850 7708 raft_consensus.cc:729] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493890a07f5f4d34b9f48a3da5421253, State: Initialized, Role: FOLLOWER
I20260809 22:53:50.679991 7708 consensus_queue.cc:260] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.683670 7706 tablet_bootstrap.cc:654] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:50.686918 7707 ts_tablet_manager.cc:1428] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060: Time spent starting tablet: real 0.046s user 0.005s sys 0.005s
I20260809 22:53:50.688328 7708 ts_tablet_manager.cc:1428] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253: Time spent starting tablet: real 0.041s user 0.013s sys 0.000s
I20260809 22:53:50.689661 7708 tablet_bootstrap.cc:492] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253: Bootstrap starting.
I20260809 22:53:50.693185 7711 raft_consensus.cc:482] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:50.693944 7711 raft_consensus.cc:504] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.699815 7711 leader_election.cc:288] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:50.702219 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:50.702884 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
W20260809 22:53:50.710780 7513 leader_election.cc:341] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533): Illegal state: must be running to vote when last-logged opid is not known
I20260809 22:53:50.706650 7644 raft_consensus.cc:2454] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 055f6d77fb794a629017dd736c1a6060 in term 0.
I20260809 22:53:50.713742 7708 tablet_bootstrap.cc:654] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253: Neither blocks nor log segments found. Creating new log.
I20260809 22:53:50.712757 7238 catalog_manager.cc:5495] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8db860deafb41209b9e337a987f2887 (127.5.105.131). New cstate: current_term: 1 leader_uuid: "d8db860deafb41209b9e337a987f2887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } health_report { overall_health: UNKNOWN } } }
I20260809 22:53:50.714551 7513 leader_election.cc:302] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters: 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:50.716024 7711 raft_consensus.cc:2781] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:53:50.716459 7711 raft_consensus.cc:482] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:53:50.716864 7711 raft_consensus.cc:3037] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:50.724013 7706 tablet_bootstrap.cc:492] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887: No bootstrap required, opened a new log
I20260809 22:53:50.725271 7706 ts_tablet_manager.cc:1397] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887: Time spent bootstrapping tablet: real 0.056s user 0.022s sys 0.005s
I20260809 22:53:50.740396 7711 raft_consensus.cc:504] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.744963 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253"
W20260809 22:53:50.746930 7513 leader_election.cc:341] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533): Illegal state: must be running to vote when last-logged opid is not known
I20260809 22:53:50.740442 7706 raft_consensus.cc:348] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.746963 7714 consensus_queue.cc:1035] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 22:53:50.748214 7706 raft_consensus.cc:374] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:50.749048 7706 raft_consensus.cc:729] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8db860deafb41209b9e337a987f2887, State: Initialized, Role: FOLLOWER
I20260809 22:53:50.750763 7711 leader_election.cc:288] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 1 election: Requested vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:50.750535 7706 consensus_queue.cc:260] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.752419 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887"
I20260809 22:53:50.753305 7644 raft_consensus.cc:3037] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:50.763507 7718 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:50.770128 7714 consensus_queue.cc:1035] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 22:53:50.773666 7706 ts_tablet_manager.cc:1428] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887: Time spent starting tablet: real 0.048s user 0.006s sys 0.006s
I20260809 22:53:50.764585 7718 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.796451 7644 raft_consensus.cc:2454] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 055f6d77fb794a629017dd736c1a6060 in term 1.
I20260809 22:53:50.800454 7513 leader_election.cc:302] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters: 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:50.805308 7708 tablet_bootstrap.cc:492] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253: No bootstrap required, opened a new log
I20260809 22:53:50.806007 7708 ts_tablet_manager.cc:1397] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253: Time spent bootstrapping tablet: real 0.117s user 0.016s sys 0.070s
I20260809 22:53:50.792567 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:50.810588 7718 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:50.809811 7714 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:50.813618 7714 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.821728 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:50.825917 7644 raft_consensus.cc:2454] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493890a07f5f4d34b9f48a3da5421253 in term 0.
I20260809 22:53:50.829012 7422 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493890a07f5f4d34b9f48a3da5421253, d8db860deafb41209b9e337a987f2887; no voters:
I20260809 22:53:50.831044 7718 raft_consensus.cc:2781] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:53:50.831594 7718 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:53:50.832147 7718 raft_consensus.cc:3037] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:50.839846 7708 raft_consensus.cc:348] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.841217 7708 raft_consensus.cc:374] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:53:50.824796 7553 raft_consensus.cc:2454] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493890a07f5f4d34b9f48a3da5421253 in term 0.
I20260809 22:53:50.841696 7708 raft_consensus.cc:729] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493890a07f5f4d34b9f48a3da5421253, State: Initialized, Role: FOLLOWER
I20260809 22:53:50.810823 7711 raft_consensus.cc:2781] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:53:50.846544 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:50.853818 7708 consensus_queue.cc:260] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.858072 7708 ts_tablet_manager.cc:1428] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253: Time spent starting tablet: real 0.052s user 0.019s sys 0.009s
I20260809 22:53:50.862941 7714 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:50.871549 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:50.872432 7553 raft_consensus.cc:2454] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8db860deafb41209b9e337a987f2887 in term 0.
I20260809 22:53:50.874172 7603 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters:
I20260809 22:53:50.886335 7716 raft_consensus.cc:482] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:50.887030 7716 raft_consensus.cc:504] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.892212 7732 raft_consensus.cc:2781] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:53:50.892787 7732 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:53:50.892976 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:50.893303 7732 raft_consensus.cc:3037] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:50.904937 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:50.907079 7644 raft_consensus.cc:2379] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 493890a07f5f4d34b9f48a3da5421253 in current term 1: Already voted for candidate 055f6d77fb794a629017dd736c1a6060 in this term.
I20260809 22:53:50.906386 7716 leader_election.cc:288] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:50.933068 7711 raft_consensus.cc:686] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 1 LEADER]: Becoming Leader. State: Replica: 055f6d77fb794a629017dd736c1a6060, State: Running, Role: LEADER
I20260809 22:53:50.939592 7711 consensus_queue.cc:237] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.946609 7718 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.942848 7732 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:50.950352 7466 raft_consensus.cc:2379] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8db860deafb41209b9e337a987f2887 in current term 1: Already voted for candidate 493890a07f5f4d34b9f48a3da5421253 in this term.
I20260809 22:53:50.955020 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060"
I20260809 22:53:50.956400 7550 raft_consensus.cc:3037] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:50.955938 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253"
I20260809 22:53:50.962779 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887"
I20260809 22:53:50.961745 7732 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 1 election: Requested vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:50.960193 7718 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 1 election: Requested vote from peers 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:50.966820 7644 raft_consensus.cc:2379] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 493890a07f5f4d34b9f48a3da5421253 in current term 1: Already voted for candidate d8db860deafb41209b9e337a987f2887 in this term.
I20260809 22:53:50.968080 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060"
I20260809 22:53:50.991278 7603 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8db860deafb41209b9e337a987f2887; no voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:50.996119 7732 raft_consensus.cc:2726] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 22:53:51.018831 7550 raft_consensus.cc:2454] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 493890a07f5f4d34b9f48a3da5421253 in term 1.
I20260809 22:53:51.024237 7426 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253; no voters: d8db860deafb41209b9e337a987f2887
I20260809 22:53:51.027654 7718 raft_consensus.cc:2781] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:53:51.028496 7718 raft_consensus.cc:686] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 1 LEADER]: Becoming Leader. State: Replica: 493890a07f5f4d34b9f48a3da5421253, State: Running, Role: LEADER
I20260809 22:53:51.042742 7467 raft_consensus.cc:3037] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:53:51.037561 7718 consensus_queue.cc:237] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:51.073320 7750 raft_consensus.cc:482] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 055f6d77fb794a629017dd736c1a6060)
I20260809 22:53:51.073977 7750 raft_consensus.cc:504] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:51.081980 7750 leader_election.cc:288] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:51.084012 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:51.089632 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:51.096294 7603 leader_election.cc:302] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8db860deafb41209b9e337a987f2887; no voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:51.105423 7732 raft_consensus.cc:2726] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 22:53:51.124852 7711 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8db860deafb41209b9e337a987f2887)
I20260809 22:53:51.125986 7711 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:51.131560 7711 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583), 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533)
I20260809 22:53:51.140421 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:51.174352 7464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:51.176757 7513 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060; no voters: 493890a07f5f4d34b9f48a3da5421253, d8db860deafb41209b9e337a987f2887
W20260809 22:53:51.177083 7554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60494 (request call id 2) took 285 ms (client timeout 217 ms). Trace:
I20260809 22:53:51.177824 7711 raft_consensus.cc:2726] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260809 22:53:51.177706 7554 rpcz_store.cc:269] 0809 22:53:50.892212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 22:53:50.892426 (+ 214us) service_pool.cc:227] Handling call
0809 22:53:51.176856 (+284430us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b5400380254. stack: 00007fe3cab13033 00007fe3cab12f30 00007fe3ca471534 00000000003b4464 00000000003b43f9 00007fe3d1337471 00007fe3d13b62ae 00007fe3d35bfc27 00007fe3d0f47c04 00007fe3d0f47b92 00007fe3d0f47ac1 00007fe3d0f47a3c 00007fe3d0f46cf2 00007fe3cf67394c 00007fe3cf672d86 00007fe3cf67272f
0809 22:53:51.177015 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":399794560}
I20260809 22:53:51.182174 7758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 75d10a05561a482f9823bab385db5ed3)
I20260809 22:53:51.182893 7758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:51.189164 7758 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015), 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689)
I20260809 22:53:51.203889 7732 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:51.204609 7732 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:51.207412 7464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:51.207942 7732 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:51.209791 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:51.210552 7554 raft_consensus.cc:2454] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8db860deafb41209b9e337a987f2887 in term 1.
I20260809 22:53:51.212090 7603 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters:
I20260809 22:53:51.213289 7732 raft_consensus.cc:2781] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 22:53:51.213893 7732 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:53:51.214490 7732 raft_consensus.cc:3037] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:51.229425 7732 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:51.233172 7732 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 2 election: Requested vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:51.235114 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253"
I20260809 22:53:51.260839 7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "23b696fe94a441ba93177a12ff465c26" is_pre_election: true
I20260809 22:53:51.261909 7172 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ed55e55e8fd4ad4830e8fd57d1d7cac in term 1.
I20260809 22:53:51.267545 7293 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac, 23b696fe94a441ba93177a12ff465c26; no voters:
W20260809 22:53:51.268040 7466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39140 (request call id 5) took 422 ms (client timeout 271 ms). Trace:
I20260809 22:53:51.268983 7758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260809 22:53:51.268633 7466 rpcz_store.cc:269] 0809 22:53:50.845576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 22:53:50.845944 (+ 368us) service_pool.cc:227] Handling call
0809 22:53:51.267708 (+421764us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b5400390154. stack: 00007fe3cab13033 00007fe3cab12f30 00007fe3ca471534 00000000003b4464 00000000003b43f9 00007fe3d1337471 00007fe3d13b62ae 00007fe3d35bfc27 00007fe3d0f47c04 00007fe3d0f47b92 00007fe3d0f47ac1 00007fe3d0f47a3c 00007fe3d0f46cf2 00007fe3cf67394c 00007fe3cf672d86 00007fe3cf67272f
0809 22:53:51.267959 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":233415936}
I20260809 22:53:51.269763 7758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 1 FOLLOWER]: Starting leader election (detected failure of leader 75d10a05561a482f9823bab385db5ed3)
I20260809 22:53:51.270530 7758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:51.280402 7758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:51.285329 7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "23b696fe94a441ba93177a12ff465c26"
I20260809 22:53:51.286291 7172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:51.287730 7758 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 2 election: Requested vote from peers 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015), 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689)
I20260809 22:53:51.290016 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "75d10a05561a482f9823bab385db5ed3"
W20260809 22:53:51.522621 7426 leader_election.cc:334] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821): Timed out: RequestConsensusVote RPC to 127.5.105.130:33821 timed out after 0.217s (SENT)
I20260809 22:53:51.523492 7426 leader_election.cc:302] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493890a07f5f4d34b9f48a3da5421253; no voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887
I20260809 22:53:51.730708 7172 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ed55e55e8fd4ad4830e8fd57d1d7cac in term 2.
W20260809 22:53:51.872783 7172 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38924 (request call id 4) took 588 ms (client timeout 239 ms). Trace:
W20260809 22:53:51.873293 7172 rpcz_store.cc:269] 0809 22:53:51.284656 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 22:53:51.284839 (+ 183us) service_pool.cc:227] Handling call
0809 22:53:51.872689 (+587850us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 22:53:51.981421 7293 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.5.105.190:34015, user_credentials={real_user=slave}} blocked reactor thread for 458058us
I20260809 22:53:51.983274 7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "23b696fe94a441ba93177a12ff465c26"
I20260809 22:53:51.983877 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "75d10a05561a482f9823bab385db5ed3" is_pre_election: true
I20260809 22:53:51.984061 7172 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1ed55e55e8fd4ad4830e8fd57d1d7cac in term 2. Re-sending same reply.
I20260809 22:53:51.985297 7293 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac, 23b696fe94a441ba93177a12ff465c26; no voters: 75d10a05561a482f9823bab385db5ed3
I20260809 22:53:51.986881 7762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 2 FOLLOWER]: Leader election won for term 2
I20260809 22:53:51.988420 7762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 2 LEADER]: Becoming Leader. State: Replica: 1ed55e55e8fd4ad4830e8fd57d1d7cac, State: Running, Role: LEADER
I20260809 22:53:51.989444 7762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:52.002835 7322 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 2 LEADER]: Rejecting Update request from peer 75d10a05561a482f9823bab385db5ed3 for earlier term 1. Current term is 2. Ops: []
W20260809 22:53:52.003860 7463 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39140 (request call id 13) took 795 ms (client timeout 264 ms). Trace:
W20260809 22:53:52.004331 7463 rpcz_store.cc:269] 0809 22:53:51.207997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 22:53:52.003393 (+795396us) service_pool.cc:214] Skipping call since client already timed out
0809 22:53:52.003623 (+ 230us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260809 22:53:52.004750 7760 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 }, Status: INVALID_TERM, Last received: 1.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 22:53:52.005540 7760 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [LEADER]: Peer 1ed55e55e8fd4ad4830e8fd57d1d7cac log is divergent from this leader: its last log entry 2.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260809 22:53:52.006057 7760 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 22:53:52.006361 7760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 75d10a05561a482f9823bab385db5ed3, State: Running, Role: LEADER
I20260809 22:53:52.006606 7238 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50914 (request call id 7) took 1335 ms. Trace:
I20260809 22:53:52.007141 7760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:52.008392 7760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:52.006968 7238 rpcz_store.cc:276] 0809 22:53:50.671170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 22:53:50.671372 (+ 202us) service_pool.cc:227] Handling call
0809 22:53:52.006540 (+1335168us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 22:53:50.739021 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 22:53:50.740113 (+ 1092us) write_op.cc:432] Acquiring schema lock in shared mode
0809 22:53:50.740162 (+ 49us) write_op.cc:435] Acquired schema lock
0809 22:53:50.740201 (+ 39us) tablet.cc:660] Decoding operations
0809 22:53:50.740729 (+ 528us) write_op.cc:620] Acquiring the partition lock for write op
0809 22:53:50.740859 (+ 130us) write_op.cc:641] Partition lock acquired for write op
0809 22:53:50.740911 (+ 52us) tablet.cc:683] Acquiring locks for 1 operations
0809 22:53:50.741213 (+ 302us) tablet.cc:699] Row locks acquired
0809 22:53:50.741259 (+ 46us) write_op.cc:260] PREPARE: finished
0809 22:53:50.741474 (+ 215us) write_op.cc:270] Start()
0809 22:53:50.741642 (+ 168us) write_op.cc:276] Timestamp: P: 1786316030741427 usec, L: 0
0809 22:53:50.741682 (+ 40us) op_driver.cc:331] REPLICATION: starting
0809 22:53:50.813088 (+ 71406us) log.cc:842] Serialized 543 byte log entry
0809 22:53:50.869378 (+ 56290us) op_driver.cc:444] REPLICATION: finished
0809 22:53:50.936305 (+ 66927us) write_op.cc:301] APPLY: starting
0809 22:53:50.937197 (+ 892us) tablet.cc:1365] starting BulkCheckPresence
0809 22:53:50.937383 (+ 186us) tablet.cc:1368] finished BulkCheckPresence
0809 22:53:50.937691 (+ 308us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 22:53:50.943576 (+ 5885us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 22:53:50.948690 (+ 5114us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0809 22:53:50.948785 (+ 95us) write_op.cc:312] APPLY: finished
0809 22:53:50.960122 (+ 11337us) log.cc:842] Serialized 22 byte log entry
0809 22:53:50.965271 (+ 5149us) write_op.cc:489] Releasing partition, row and schema locks
0809 22:53:50.966375 (+ 1104us) write_op.cc:454] Released schema lock
0809 22:53:50.967237 (+ 862us) write_op.cc:341] FINISH: Updating metrics
0809 22:53:50.973039 (+ 5802us) write_op.cc:489] Releasing partition, row and schema locks
0809 22:53:50.983823 (+ 10784us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20172,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":53531,"apply.run_cpu_time_us":3527,"apply.run_wall_time_us":52020,"num_ops":1,"prepare.queue_time_us":6723,"prepare.run_cpu_time_us":5142,"prepare.run_wall_time_us":77847,"replication_time_us":126459,"thread_start_us":35350,"threads_started":2}]]}
I20260809 22:53:52.019212 7711 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8db860deafb41209b9e337a987f2887)
I20260809 22:53:52.019877 7711 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:52.022042 7711 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583), 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533)
I20260809 22:53:52.043920 7765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:52.044579 7765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:52.046463 7172 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 2 FOLLOWER]: Rejecting Update request from peer 75d10a05561a482f9823bab385db5ed3 for earlier term 1. Current term is 2. Ops: []
I20260809 22:53:52.049629 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b696fe94a441ba93177a12ff465c26" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "75d10a05561a482f9823bab385db5ed3" is_pre_election: true
I20260809 22:53:52.050520 7765 leader_election.cc:288] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689), 1ed55e55e8fd4ad4830e8fd57d1d7cac (127.5.105.188:45511)
I20260809 22:53:52.050285 7322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b696fe94a441ba93177a12ff465c26" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" is_pre_election: true
I20260809 22:53:52.050706 7248 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b696fe94a441ba93177a12ff465c26 in term 2.
I20260809 22:53:52.052318 7145 leader_election.cc:302] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b696fe94a441ba93177a12ff465c26, 75d10a05561a482f9823bab385db5ed3; no voters:
I20260809 22:53:52.053577 7765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 22:53:52.054045 7765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:53:52.054452 7765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:52.063829 7765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:52.069451 7765 leader_election.cc:288] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 3 election: Requested vote from peers 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689), 1ed55e55e8fd4ad4830e8fd57d1d7cac (127.5.105.188:45511)
I20260809 22:53:52.069859 7322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b696fe94a441ba93177a12ff465c26" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac"
I20260809 22:53:52.071084 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b696fe94a441ba93177a12ff465c26" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "75d10a05561a482f9823bab385db5ed3"
I20260809 22:53:52.071728 7248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:52.075354 7743 raft_consensus.cc:482] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 055f6d77fb794a629017dd736c1a6060)
I20260809 22:53:52.077037 7743 raft_consensus.cc:504] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:52.078843 7743 leader_election.cc:288] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:52.080116 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:52.080847 7463 raft_consensus.cc:2454] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 055f6d77fb794a629017dd736c1a6060 in term 1.
I20260809 22:53:52.084440 7467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39132 (request call id 5) took 1008 ms. Trace:
I20260809 22:53:52.084800 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:52.085523 7465 raft_consensus.cc:2454] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8db860deafb41209b9e337a987f2887 in term 1.
I20260809 22:53:52.084852 7467 rpcz_store.cc:276] 0809 22:53:51.076409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 22:53:51.080292 (+ 3883us) service_pool.cc:227] Handling call
0809 22:53:51.082924 (+ 2632us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 22:53:51.083119 (+ 195us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 22:53:51.083146 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 22:53:52.060802 (+977656us) log.cc:842] Serialized 26 byte log entry
0809 22:53:52.084077 (+ 23275us) raft_consensus.cc:1631] Marking committed up to 0
0809 22:53:52.084119 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 22:53:52.084199 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 22:53:52.084245 (+ 46us) raft_consensus.cc:1671] finished
0809 22:53:52.084258 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 22:53:52.084385 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":14912,"threads_started":1,"wal-append.queue_time_us":2636}
I20260809 22:53:52.087210 7718 raft_consensus.cc:2726] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 22:53:52.087702 7602 leader_election.cc:302] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493890a07f5f4d34b9f48a3da5421253, d8db860deafb41209b9e337a987f2887; no voters:
I20260809 22:53:52.088109 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:52.089176 7513 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253; no voters:
I20260809 22:53:52.089733 7743 raft_consensus.cc:2781] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 22:53:52.090297 7743 raft_consensus.cc:482] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 055f6d77fb794a629017dd736c1a6060)
I20260809 22:53:52.090740 7743 raft_consensus.cc:3037] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:52.091058 7711 raft_consensus.cc:2781] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 22:53:52.091495 7711 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8db860deafb41209b9e337a987f2887)
I20260809 22:53:52.099980 7746 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:52.093520 7711 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:52.100656 7746 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:52.094895 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253"
I20260809 22:53:52.117872 7746 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:52.119318 7743 raft_consensus.cc:504] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:52.121764 7718 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8db860deafb41209b9e337a987f2887)
I20260809 22:53:52.122447 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060"
W20260809 22:53:52.122903 7249 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50896 (request call id 3) took 933 ms (client timeout 279 ms). Trace:
W20260809 22:53:52.123598 7249 rpcz_store.cc:269] 0809 22:53:51.189436 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 22:53:52.122631 (+933195us) service_pool.cc:214] Skipping call since client already timed out
0809 22:53:52.122821 (+ 190us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260809 22:53:52.124930 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253"
I20260809 22:53:52.125236 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060"
I20260809 22:53:52.124653 7743 leader_election.cc:288] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 2 election: Requested vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:52.126452 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:52.122298 7711 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:52.128525 7554 raft_consensus.cc:1226] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 2 FOLLOWER]: Rejecting Update request from peer d8db860deafb41209b9e337a987f2887 for earlier term 1. Current term is 2. Ops: []
I20260809 22:53:52.128181 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:52.131228 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060"
I20260809 22:53:52.131775 7711 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 2 election: Requested vote from peers d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583), 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533)
I20260809 22:53:52.123622 7553 raft_consensus.cc:3037] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:52.135334 7603 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8db860deafb41209b9e337a987f2887; no voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:52.185707 7732 raft_consensus.cc:2726] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 22:53:52.187577 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:52.187772 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887"
I20260809 22:53:52.188783 7644 raft_consensus.cc:2379] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 055f6d77fb794a629017dd736c1a6060 in current term 2: Already voted for candidate d8db860deafb41209b9e337a987f2887 in this term.
W20260809 22:53:52.190351 7603 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.5.105.130:33821, user_credentials={real_user=slave}} blocked reactor thread for 55210.6us
I20260809 22:53:52.156225 7763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ed55e55e8fd4ad4830e8fd57d1d7cac. Latest consensus state: current_term: 2 leader_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:52.181684 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:52.193823 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:52.197575 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253"
I20260809 22:53:52.198666 7553 raft_consensus.cc:2454] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8db860deafb41209b9e337a987f2887 in term 2.
I20260809 22:53:52.199898 7549 raft_consensus.cc:1226] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 2 FOLLOWER]: Rejecting Update request from peer 493890a07f5f4d34b9f48a3da5421253 for earlier term 1. Current term is 2. Ops: []
I20260809 22:53:52.132959 7161 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38946 (request call id 13) took 1003 ms. Trace:
I20260809 22:53:52.202713 7763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:52.205933 7603 leader_election.cc:302] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8db860deafb41209b9e337a987f2887; no voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:52.207059 7644 raft_consensus.cc:1226] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Rejecting Update request from peer 493890a07f5f4d34b9f48a3da5421253 for earlier term 1. Current term is 2. Ops: []
I20260809 22:53:52.145099 7718 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:52.208158 7732 raft_consensus.cc:2726] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 22:53:52.130275 7248 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b696fe94a441ba93177a12ff465c26 in term 3.
I20260809 22:53:52.210119 7732 consensus_queue.cc:1046] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 22:53:52.211169 7732 raft_consensus.cc:3032] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 22:53:52.211516 7732 raft_consensus.cc:729] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8db860deafb41209b9e337a987f2887, State: Running, Role: LEADER
I20260809 22:53:52.213104 7732 consensus_queue.cc:260] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:52.203117 7161 rpcz_store.cc:276] 0809 22:53:51.130079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 22:53:52.130384 (+1000305us) service_pool.cc:227] Handling call
0809 22:53:52.132881 (+ 2497us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 22:53:52.215643 7247 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 3 FOLLOWER]: Rejecting Update request from peer 1ed55e55e8fd4ad4830e8fd57d1d7cac for earlier term 2. Current term is 3. Ops: []
I20260809 22:53:52.216125 7466 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:52.217240 7716 consensus_queue.cc:1046] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 22:53:52.221773 7236 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50910 (request call id 13) took 1369 ms. Trace:
I20260809 22:53:52.222242 7236 rpcz_store.cc:276] 0809 22:53:50.852364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 22:53:50.852550 (+ 186us) service_pool.cc:227] Handling call
0809 22:53:52.221695 (+1369145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1074,"reported_tablets":1}
I20260809 22:53:52.234593 7732 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:52.239216 7173 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 3 FOLLOWER]: Rejecting Update request from peer 1ed55e55e8fd4ad4830e8fd57d1d7cac for earlier term 2. Current term is 3. Ops: []
W20260809 22:53:52.242259 7685 master_proxy_rpc.cc:203] Re-attempting IsCreateTableDone request to leader Master (127.5.105.189:33689)
I20260809 22:53:52.242609 7762 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 22:53:52.246776 7145 leader_election.cc:302] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23b696fe94a441ba93177a12ff465c26, 75d10a05561a482f9823bab385db5ed3; no voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac
I20260809 22:53:52.248281 7765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 22:53:52.249202 7718 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:52.251386 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
W20260809 22:53:52.265034 7493 debug-util.cc:398] Leaking SignalData structure 0x7b0800199a80 after lost signal to thread 5546
I20260809 22:53:52.266310 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:52.268435 7237 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50900 (request call id 13) took 1446 ms. Trace:
I20260809 22:53:52.269165 7237 rpcz_store.cc:276] 0809 22:53:50.805846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 22:53:50.805991 (+ 145us) service_pool.cc:227] Handling call
0809 22:53:50.818725 (+ 12734us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b540006e854. stack: 00007fe3cab13033 00007fe3cab12f30 00007fe3ca471534 00000000003b4464 00000000003b43f9 00007fe3d1337af0 00007fe3d13baf2a 00007fe3d39d3d6c 00007fe3d3acdc3e 00007fe3d1dd9fb4 00007fe3d1dd9f42 00007fe3d1dd9e71 00007fe3d1dd9dec 00007fe3d1dd90a2 00007fe3cf67394c 00007fe3cf672d86
0809 22:53:52.251901 (+1433176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28877,"reported_tablets":1,"spinlock_wait_cycles":23217536}
I20260809 22:53:52.261085 7762 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 2 LEADER]: Stepping down as leader of term 2
I20260809 22:53:52.228873 7513 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060; no voters: 493890a07f5f4d34b9f48a3da5421253, d8db860deafb41209b9e337a987f2887
W20260809 22:53:52.276557 7493 debug-util.cc:398] Leaking SignalData structure 0x7b08000ec540 after lost signal to thread 7199
W20260809 22:53:52.282236 7493 debug-util.cc:398] Leaking SignalData structure 0x7b0800169540 after lost signal to thread 7275
I20260809 22:53:52.284150 7711 raft_consensus.cc:2726] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260809 22:53:52.284212 7493 debug-util.cc:398] Leaking SignalData structure 0x7b080023fc60 after lost signal to thread 7350
W20260809 22:53:52.285588 7513 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.5.105.129:33533, user_credentials={real_user=slave}} blocked reactor thread for 84747.6us
I20260809 22:53:52.288337 7645 raft_consensus.cc:1226] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Rejecting Update request from peer 055f6d77fb794a629017dd736c1a6060 for earlier term 1. Current term is 2. Ops: []
I20260809 22:53:52.278077 7762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1ed55e55e8fd4ad4830e8fd57d1d7cac, State: Running, Role: LEADER
W20260809 22:53:52.290912 7493 debug-util.cc:398] Leaking SignalData structure 0x7b0800234300 after lost signal to thread 7496
W20260809 22:53:52.292469 7493 debug-util.cc:398] Leaking SignalData structure 0x7b0800234b80 after lost signal to thread 7582
I20260809 22:53:52.278146 7549 raft_consensus.cc:2379] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 493890a07f5f4d34b9f48a3da5421253 in current term 2: Already voted for candidate 055f6d77fb794a629017dd736c1a6060 in this term.
W20260809 22:53:52.294664 7493 debug-util.cc:398] Leaking SignalData structure 0x7b0800157240 after lost signal to thread 7671
I20260809 22:53:52.291400 7762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.8, Last appended by leader: 8, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:52.301640 7466 raft_consensus.cc:2454] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 055f6d77fb794a629017dd736c1a6060 in term 2.
I20260809 22:53:52.275705 7716 consensus_queue.cc:1046] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20260809 22:53:52.263932 7765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 3 LEADER]: Becoming Leader. State: Replica: 23b696fe94a441ba93177a12ff465c26, State: Running, Role: LEADER
I20260809 22:53:52.305933 7711 consensus_queue.cc:1046] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 22:53:52.307971 7746 raft_consensus.cc:3032] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 22:53:52.308461 7746 raft_consensus.cc:729] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 055f6d77fb794a629017dd736c1a6060, State: Running, Role: LEADER
I20260809 22:53:52.309581 7746 consensus_queue.cc:260] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:52.311076 7746 raft_consensus.cc:3037] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:52.316679 7644 raft_consensus.cc:2454] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493890a07f5f4d34b9f48a3da5421253 in term 2.
I20260809 22:53:52.323109 7422 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 493890a07f5f4d34b9f48a3da5421253, d8db860deafb41209b9e337a987f2887; no voters: 055f6d77fb794a629017dd736c1a6060
I20260809 22:53:52.327517 7771 raft_consensus.cc:2741] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260809 22:53:52.328220 7718 raft_consensus.cc:3032] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 22:53:52.326370 7765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:52.328801 7718 raft_consensus.cc:729] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 493890a07f5f4d34b9f48a3da5421253, State: Running, Role: LEADER
I20260809 22:53:52.329859 7718 consensus_queue.cc:260] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:52.333777 7549 raft_consensus.cc:1226] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 2 FOLLOWER]: Rejecting Update request from peer 493890a07f5f4d34b9f48a3da5421253 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260809 22:53:52.340641 7773 consensus_queue.cc:1046] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 22:53:52.342331 7718 raft_consensus.cc:3037] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:53:52.343148 7644 raft_consensus.cc:1226] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Rejecting Update request from peer 493890a07f5f4d34b9f48a3da5421253 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260809 22:53:52.343267 7774 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260809 22:53:52.345901 7463 raft_consensus.cc:1226] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 2 FOLLOWER]: Rejecting Update request from peer d8db860deafb41209b9e337a987f2887 for earlier term 1. Current term is 2. Ops: []
I20260809 22:53:52.347483 7762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:52.348637 7513 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253; no voters: d8db860deafb41209b9e337a987f2887
I20260809 22:53:52.350904 7746 raft_consensus.cc:2781] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 22:53:52.354568 7746 raft_consensus.cc:686] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 2 LEADER]: Becoming Leader. State: Replica: 055f6d77fb794a629017dd736c1a6060, State: Running, Role: LEADER
I20260809 22:53:52.356503 7746 consensus_queue.cc:237] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:52.360639 7775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b696fe94a441ba93177a12ff465c26. Latest consensus state: current_term: 3 leader_uuid: "23b696fe94a441ba93177a12ff465c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:52.361912 7775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:52.385785 7730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:52.386690 7730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
W20260809 22:53:52.396484 7776 catalog_manager.cc:1409] Failed waiting for node to catch up after master election: Illegal state: Node 1ed55e55e8fd4ad4830e8fd57d1d7cac not leader. Raft Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:52.403776 7730 leader_election.cc:288] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015), 1ed55e55e8fd4ad4830e8fd57d1d7cac (127.5.105.188:45511)
I20260809 22:53:52.405637 7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75d10a05561a482f9823bab385db5ed3" candidate_term: 4 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "23b696fe94a441ba93177a12ff465c26" is_pre_election: true
I20260809 22:53:52.407229 7322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75d10a05561a482f9823bab385db5ed3" candidate_term: 4 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" is_pre_election: true
I20260809 22:53:52.408213 7322 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75d10a05561a482f9823bab385db5ed3 for term 4 because replica has last-logged OpId of term: 2 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 7.
I20260809 22:53:52.411340 7218 leader_election.cc:302] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75d10a05561a482f9823bab385db5ed3; no voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac, 23b696fe94a441ba93177a12ff465c26
I20260809 22:53:52.412436 7778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:53:52.413739 7730 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 22:53:52.427932 7778 catalog_manager.cc:669] Loaded metadata for table test-table [id=842a76a4c8694fd187d3c04383286ae1]
I20260809 22:53:52.437965 7732 raft_consensus.cc:482] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:52.438815 7732 raft_consensus.cc:504] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:52.440429 7778 tablet_loader.cc:96] loaded metadata for tablet 119c8382d78c4413aefb2d2d9077d47e (table test-table [id=842a76a4c8694fd187d3c04383286ae1])
I20260809 22:53:52.442739 7778 tablet_loader.cc:96] loaded metadata for tablet 19ef6c21cc3c497483e1ddb9b9fe17ef (table test-table [id=842a76a4c8694fd187d3c04383286ae1])
I20260809 22:53:52.442935 7732 leader_election.cc:288] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:52.443718 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:52.444916 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:52.445801 7549 raft_consensus.cc:2454] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8db860deafb41209b9e337a987f2887 in term 2.
I20260809 22:53:52.448212 7778 tablet_loader.cc:96] loaded metadata for tablet 2ad6955771cc4caa8eda877ded5a808a (table test-table [id=842a76a4c8694fd187d3c04383286ae1])
I20260809 22:53:52.450436 7322 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 3 FOLLOWER]: Refusing update from remote peer 23b696fe94a441ba93177a12ff465c26: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 8. (term mismatch)
I20260809 22:53:52.451007 7322 pending_rounds.cc:76] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac: Aborting all ops after (but not including) 7
I20260809 22:53:52.451464 7322 pending_rounds.cc:97] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac: Aborting uncommitted NO_OP operation due to leader change: 2.8
I20260809 22:53:52.451542 7603 leader_election.cc:302] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters: 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:52.452116 7322 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 22:53:52.452243 7247 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 3 FOLLOWER]: Refusing update from remote peer 23b696fe94a441ba93177a12ff465c26: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20260809 22:53:52.453323 7732 raft_consensus.cc:2781] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 22:53:52.453919 7732 raft_consensus.cc:482] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:53:52.454806 7732 raft_consensus.cc:3037] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:52.450376 7778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:53:52.455163 7765 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 22:53:52.456321 7775 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 22:53:52.464031 7778 catalog_manager.cc:1259] Loaded cluster ID: f511035ecb4f4aaa9b8234496968ea8e
I20260809 22:53:52.466897 7644 raft_consensus.cc:1261] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Refusing update from remote peer 055f6d77fb794a629017dd736c1a6060: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 22:53:52.468248 7778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:53:52.470536 7746 consensus_queue.cc:1035] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 22:53:52.472385 7732 raft_consensus.cc:504] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:52.473598 7762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b696fe94a441ba93177a12ff465c26. Latest consensus state: current_term: 3 leader_uuid: "23b696fe94a441ba93177a12ff465c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:52.482352 7762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:52.478995 7463 raft_consensus.cc:1261] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 2 FOLLOWER]: Refusing update from remote peer 055f6d77fb794a629017dd736c1a6060: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 22:53:52.488256 7746 consensus_queue.cc:1035] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260809 22:53:52.495147 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253"
I20260809 22:53:52.502816 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060"
I20260809 22:53:52.503752 7549 raft_consensus.cc:3037] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:52.498620 7778 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:53:52.507830 7732 leader_election.cc:288] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 3 election: Requested vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:52.528023 7730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b696fe94a441ba93177a12ff465c26. Latest consensus state: current_term: 3 leader_uuid: "23b696fe94a441ba93177a12ff465c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:52.529242 7730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:52.530624 7718 raft_consensus.cc:482] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 055f6d77fb794a629017dd736c1a6060)
I20260809 22:53:52.531778 7718 raft_consensus.cc:504] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:52.522693 7732 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:52.538278 7765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "23b696fe94a441ba93177a12ff465c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:52.540159 7765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:52.540335 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:52.540736 7732 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:52.543087 7732 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:52.545408 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:52.547041 7466 raft_consensus.cc:2396] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8db860deafb41209b9e337a987f2887 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260809 22:53:52.548197 7778 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: Loaded TSK: 0
I20260809 22:53:52.550803 7778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:53:52.552641 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:52.553988 7645 raft_consensus.cc:2354] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 493890a07f5f4d34b9f48a3da5421253 for earlier term 2. Current term is 3.
I20260809 22:53:52.555430 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:52.556447 7550 raft_consensus.cc:2454] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8db860deafb41209b9e337a987f2887 in term 2.
I20260809 22:53:52.558224 7603 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters: 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:52.559571 7732 raft_consensus.cc:2781] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 22:53:52.559832 7732 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:53:52.560029 7732 raft_consensus.cc:3037] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:52.562981 7718 leader_election.cc:288] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:52.564621 7422 leader_election.cc:398] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583) with higher term. Message: Invalid argument: T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 493890a07f5f4d34b9f48a3da5421253 for earlier term 2. Current term is 3.
I20260809 22:53:52.565169 7422 leader_election.cc:401] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260809 22:53:52.566998 7716 raft_consensus.cc:3037] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 1 FOLLOWER]: Advancing to term 3
I20260809 22:53:52.567776 7718 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:52.580644 7762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "23b696fe94a441ba93177a12ff465c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:52.582533 7762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:52.584928 7160 catalog_manager.cc:5495] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 reported cstate change: term changed from 1 to 2, leader changed from d8db860deafb41209b9e337a987f2887 (127.5.105.131) to 055f6d77fb794a629017dd736c1a6060 (127.5.105.130). New cstate: current_term: 2 leader_uuid: "055f6d77fb794a629017dd736c1a6060" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } }
I20260809 22:53:52.610071 7549 raft_consensus.cc:2454] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8db860deafb41209b9e337a987f2887 in term 3.
I20260809 22:53:52.611891 7603 leader_election.cc:302] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters: 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:52.613013 7732 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:52.613528 7743 raft_consensus.cc:2781] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 22:53:52.591095 7718 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:52.614892 7765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "23b696fe94a441ba93177a12ff465c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:52.615911 7765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:52.619540 7718 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:52.621616 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:52.622818 7549 raft_consensus.cc:2454] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493890a07f5f4d34b9f48a3da5421253 in term 2.
I20260809 22:53:52.624428 7732 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 3 election: Requested vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:52.625890 7426 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253; no voters:
I20260809 22:53:52.647425 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060"
I20260809 22:53:52.648351 7549 raft_consensus.cc:3037] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:52.648574 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253"
I20260809 22:53:52.627528 7718 raft_consensus.cc:2781] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 22:53:52.653203 7718 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:53:52.653604 7718 raft_consensus.cc:3037] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:52.655059 7586 heartbeater.cc:503] Master 127.5.105.190:34015 was elected leader, sending a full tablet report...
I20260809 22:53:52.670130 7716 raft_consensus.cc:2726] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583) with higher term. Message: Invalid argument: T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 493890a07f5f4d34b9f48a3da5421253 for earlier term 2. Current term is 3.
I20260809 22:53:52.656905 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:52.688406 7644 raft_consensus.cc:2379] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 493890a07f5f4d34b9f48a3da5421253 in current term 3: Already voted for candidate d8db860deafb41209b9e337a987f2887 in this term.
I20260809 22:53:52.689042 7730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "23b696fe94a441ba93177a12ff465c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:52.690042 7730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:52.679389 7743 raft_consensus.cc:686] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 3 LEADER]: Becoming Leader. State: Replica: d8db860deafb41209b9e337a987f2887, State: Running, Role: LEADER
I20260809 22:53:52.708482 7718 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:52.711468 7466 raft_consensus.cc:2379] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8db860deafb41209b9e337a987f2887 in current term 3: Already voted for candidate 493890a07f5f4d34b9f48a3da5421253 in this term.
I20260809 22:53:52.717816 7743 consensus_queue.cc:237] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:52.720633 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060"
I20260809 22:53:52.723958 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887"
I20260809 22:53:52.725242 7644 raft_consensus.cc:2379] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 493890a07f5f4d34b9f48a3da5421253 in current term 3: Already voted for candidate d8db860deafb41209b9e337a987f2887 in this term.
I20260809 22:53:52.726907 7422 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493890a07f5f4d34b9f48a3da5421253; no voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887
I20260809 22:53:52.730767 7716 raft_consensus.cc:2726] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 22:53:52.728278 7718 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 3 election: Requested vote from peers 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:52.745685 7549 raft_consensus.cc:2454] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8db860deafb41209b9e337a987f2887 in term 3.
I20260809 22:53:52.761544 7499 heartbeater.cc:503] Master 127.5.105.190:34015 was elected leader, sending a full tablet report...
I20260809 22:53:52.762836 7603 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters: 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:52.764611 7791 raft_consensus.cc:2781] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 22:53:52.765436 7791 raft_consensus.cc:686] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 3 LEADER]: Becoming Leader. State: Replica: d8db860deafb41209b9e337a987f2887, State: Running, Role: LEADER
I20260809 22:53:52.766738 7791 consensus_queue.cc:237] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:52.769984 7675 heartbeater.cc:503] Master 127.5.105.190:34015 was elected leader, sending a full tablet report...
I20260809 22:53:52.793555 7161 catalog_manager.cc:5495] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 reported cstate change: term changed from 0 to 3, leader changed from <none> to d8db860deafb41209b9e337a987f2887 (127.5.105.131). New cstate: current_term: 3 leader_uuid: "d8db860deafb41209b9e337a987f2887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } health_report { overall_health: HEALTHY } } }
I20260809 22:53:52.797349 7161 catalog_manager.cc:5495] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 reported cstate change: term changed from 0 to 3, leader changed from <none> to d8db860deafb41209b9e337a987f2887 (127.5.105.131). New cstate: current_term: 3 leader_uuid: "d8db860deafb41209b9e337a987f2887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } health_report { overall_health: HEALTHY } } }
I20260809 22:53:52.811156 7466 raft_consensus.cc:1261] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Refusing update from remote peer d8db860deafb41209b9e337a987f2887: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 22:53:52.813620 7743 consensus_queue.cc:1035] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 22:53:52.835377 7549 raft_consensus.cc:1261] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Refusing update from remote peer d8db860deafb41209b9e337a987f2887: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 22:53:52.840533 7732 consensus_queue.cc:1035] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 22:53:52.874281 7466 raft_consensus.cc:1261] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Refusing update from remote peer d8db860deafb41209b9e337a987f2887: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 1. (term mismatch)
I20260809 22:53:52.874833 7466 pending_rounds.cc:76] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253: Aborting all ops after (but not including) 0
I20260809 22:53:52.875166 7466 pending_rounds.cc:97] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260809 22:53:52.875557 7466 raft_consensus.cc:2866] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 22:53:52.878868 7732 consensus_queue.cc:1035] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 22:53:52.895449 7549 raft_consensus.cc:1261] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Refusing update from remote peer d8db860deafb41209b9e337a987f2887: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch)
I20260809 22:53:52.897624 7732 consensus_queue.cc:1035] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 22:53:53.068053 7730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 23b696fe94a441ba93177a12ff465c26)
I20260809 22:53:53.068833 7730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:53.073704 7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75d10a05561a482f9823bab385db5ed3" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "23b696fe94a441ba93177a12ff465c26" is_pre_election: true
I20260809 22:53:53.078747 7322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75d10a05561a482f9823bab385db5ed3" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" is_pre_election: true
I20260809 22:53:53.081108 7218 leader_election.cc:302] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75d10a05561a482f9823bab385db5ed3; no voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac, 23b696fe94a441ba93177a12ff465c26
I20260809 22:53:53.083343 7730 leader_election.cc:288] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015), 1ed55e55e8fd4ad4830e8fd57d1d7cac (127.5.105.188:45511)
I20260809 22:53:53.090292 7730 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 22:53:53.126515 7218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.5.105.188:45511, user_credentials={real_user=slave}} blocked reactor thread for 45534.4us
I20260809 22:53:53.218617 7372 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 22:53:54.238884 7550 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef"
dest_uuid: "055f6d77fb794a629017dd736c1a6060"
from {username='slave'} at 127.0.0.1:39226
I20260809 22:53:54.239934 7550 raft_consensus.cc:580] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 2 LEADER]: Received request to step down
I20260809 22:53:54.240422 7550 raft_consensus.cc:3032] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 22:53:54.240744 7550 raft_consensus.cc:729] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 055f6d77fb794a629017dd736c1a6060, State: Running, Role: LEADER
I20260809 22:53:54.241588 7550 consensus_queue.cc:260] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:54.243141 7550 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:54.386121 7804 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 055f6d77fb794a629017dd736c1a6060)
I20260809 22:53:54.386852 7804 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:54.389153 7804 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:54.395296 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:54.398355 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:54.399204 7550 raft_consensus.cc:2454] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8db860deafb41209b9e337a987f2887 in term 3.
I20260809 22:53:54.400689 7603 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters: 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:54.401628 7732 raft_consensus.cc:2781] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 22:53:54.401990 7732 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 055f6d77fb794a629017dd736c1a6060)
I20260809 22:53:54.402336 7732 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:54.408008 7732 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:54.409631 7732 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 3 election: Requested vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:54.410530 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253"
I20260809 22:53:54.410717 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060"
I20260809 22:53:54.415648 7550 raft_consensus.cc:2454] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8db860deafb41209b9e337a987f2887 in term 3.
I20260809 22:53:54.416635 7603 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters: 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:54.417440 7732 raft_consensus.cc:2781] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 22:53:54.417906 7732 raft_consensus.cc:686] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 3 LEADER]: Becoming Leader. State: Replica: d8db860deafb41209b9e337a987f2887, State: Running, Role: LEADER
I20260809 22:53:54.418781 7732 consensus_queue.cc:237] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:54.428840 7160 catalog_manager.cc:5495] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 reported cstate change: term changed from 2 to 3, leader changed from 055f6d77fb794a629017dd736c1a6060 (127.5.105.130) to d8db860deafb41209b9e337a987f2887 (127.5.105.131). New cstate: current_term: 3 leader_uuid: "d8db860deafb41209b9e337a987f2887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } health_report { overall_health: UNKNOWN } } }
I20260809 22:53:54.435316 7837 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 055f6d77fb794a629017dd736c1a6060)
I20260809 22:53:54.435871 7837 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:54.440032 7837 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:54.440157 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:54.440961 7550 raft_consensus.cc:2379] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 493890a07f5f4d34b9f48a3da5421253 in current term 3: Already voted for candidate d8db860deafb41209b9e337a987f2887 in this term.
I20260809 22:53:54.441023 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:54.443739 7426 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493890a07f5f4d34b9f48a3da5421253; no voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887
I20260809 22:53:54.452903 7837 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:53:54.476235 7837 raft_consensus.cc:2726] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 22:53:54.499207 7644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e"
dest_uuid: "d8db860deafb41209b9e337a987f2887"
from {username='slave'} at 127.0.0.1:47068
I20260809 22:53:54.499840 7644 raft_consensus.cc:580] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 3 LEADER]: Received request to step down
I20260809 22:53:54.500181 7644 raft_consensus.cc:3032] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 22:53:54.500480 7644 raft_consensus.cc:729] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8db860deafb41209b9e337a987f2887, State: Running, Role: LEADER
I20260809 22:53:54.501230 7644 consensus_queue.cc:260] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:54.502741 7644 raft_consensus.cc:3037] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 3 FOLLOWER]: Advancing to term 4
I20260809 22:53:54.510572 7550 raft_consensus.cc:1261] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Refusing update from remote peer d8db860deafb41209b9e337a987f2887: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 22:53:54.511945 7463 raft_consensus.cc:1261] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Refusing update from remote peer d8db860deafb41209b9e337a987f2887: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 22:53:54.513584 7732 consensus_queue.cc:1035] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 22:53:54.517406 7732 consensus_queue.cc:1035] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 22:53:54.623247 7711 raft_consensus.cc:482] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8db860deafb41209b9e337a987f2887)
I20260809 22:53:54.623858 7711 raft_consensus.cc:504] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:54.626107 7711 leader_election.cc:288] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:54.626988 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:54.627247 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:54.628031 7644 raft_consensus.cc:2454] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 055f6d77fb794a629017dd736c1a6060 in term 4.
I20260809 22:53:54.629135 7513 leader_election.cc:302] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters: 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:54.630081 7711 raft_consensus.cc:2781] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 22:53:54.630441 7711 raft_consensus.cc:482] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8db860deafb41209b9e337a987f2887)
I20260809 22:53:54.630806 7711 raft_consensus.cc:3037] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Advancing to term 4
I20260809 22:53:54.650821 7711 raft_consensus.cc:504] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:54.653003 7711 leader_election.cc:288] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 4 election: Requested vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:54.654156 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253"
I20260809 22:53:54.654520 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887"
I20260809 22:53:54.681174 7837 raft_consensus.cc:482] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8db860deafb41209b9e337a987f2887)
I20260809 22:53:54.681756 7837 raft_consensus.cc:504] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:54.684365 7837 leader_election.cc:288] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:54.686049 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:54.687505 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:54.688316 7549 raft_consensus.cc:2379] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 493890a07f5f4d34b9f48a3da5421253 in current term 4: Already voted for candidate 055f6d77fb794a629017dd736c1a6060 in this term.
I20260809 22:53:54.689329 7426 leader_election.cc:302] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493890a07f5f4d34b9f48a3da5421253; no voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887
I20260809 22:53:54.690375 7837 raft_consensus.cc:3037] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Advancing to term 4
I20260809 22:53:54.722519 7645 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a"
dest_uuid: "d8db860deafb41209b9e337a987f2887"
from {username='slave'} at 127.0.0.1:47068
I20260809 22:53:54.723232 7645 raft_consensus.cc:580] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 3 LEADER]: Received request to step down
I20260809 22:53:54.723624 7645 raft_consensus.cc:3032] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 22:53:54.723914 7645 raft_consensus.cc:729] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8db860deafb41209b9e337a987f2887, State: Running, Role: LEADER
I20260809 22:53:54.724637 7645 consensus_queue.cc:260] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:54.725957 7645 raft_consensus.cc:3037] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 3 FOLLOWER]: Advancing to term 4
I20260809 22:53:54.751796 7644 raft_consensus.cc:2454] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 055f6d77fb794a629017dd736c1a6060 in term 4.
I20260809 22:53:54.752009 7837 raft_consensus.cc:2726] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 22:53:54.755762 7513 leader_election.cc:302] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters: 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:54.757216 7711 raft_consensus.cc:2781] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 22:53:54.758728 7711 raft_consensus.cc:686] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 4 LEADER]: Becoming Leader. State: Replica: 055f6d77fb794a629017dd736c1a6060, State: Running, Role: LEADER
I20260809 22:53:54.759933 7711 consensus_queue.cc:237] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:54.768301 7153 catalog_manager.cc:5495] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 reported cstate change: term changed from 3 to 4, leader changed from d8db860deafb41209b9e337a987f2887 (127.5.105.131) to 055f6d77fb794a629017dd736c1a6060 (127.5.105.130). New cstate: current_term: 4 leader_uuid: "055f6d77fb794a629017dd736c1a6060" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 22:53:54.839543 7644 raft_consensus.cc:1261] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Refusing update from remote peer 055f6d77fb794a629017dd736c1a6060: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 22:53:54.840884 7711 consensus_queue.cc:1035] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 22:53:54.853884 7463 raft_consensus.cc:1261] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Refusing update from remote peer 055f6d77fb794a629017dd736c1a6060: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 22:53:54.859616 7711 consensus_queue.cc:1035] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 22:53:54.886672 7711 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8db860deafb41209b9e337a987f2887)
I20260809 22:53:54.887334 7711 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:54.889446 7711 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:54.890961 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:54.891683 7644 raft_consensus.cc:2454] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 055f6d77fb794a629017dd736c1a6060 in term 4.
I20260809 22:53:54.892859 7513 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters:
I20260809 22:53:54.893684 7711 raft_consensus.cc:2781] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 22:53:54.894025 7711 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8db860deafb41209b9e337a987f2887)
I20260809 22:53:54.894377 7711 raft_consensus.cc:3037] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Advancing to term 4
I20260809 22:53:54.895107 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:54.909046 7853 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8db860deafb41209b9e337a987f2887)
I20260809 22:53:54.909708 7853 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:54.913038 7853 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:54.914526 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:54.915194 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:54.915934 7644 raft_consensus.cc:2454] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493890a07f5f4d34b9f48a3da5421253 in term 4.
I20260809 22:53:54.917623 7422 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493890a07f5f4d34b9f48a3da5421253, d8db860deafb41209b9e337a987f2887; no voters:
I20260809 22:53:54.918757 7853 raft_consensus.cc:2781] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 22:53:54.919190 7853 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8db860deafb41209b9e337a987f2887)
I20260809 22:53:54.919598 7853 raft_consensus.cc:3037] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Advancing to term 4
I20260809 22:53:54.932009 7644 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef"
dest_uuid: "d8db860deafb41209b9e337a987f2887"
from {username='slave'} at 127.0.0.1:47068
I20260809 22:53:54.933040 7644 raft_consensus.cc:580] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 3 LEADER]: Received request to step down
I20260809 22:53:54.933553 7644 raft_consensus.cc:3032] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 22:53:54.934024 7644 raft_consensus.cc:729] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8db860deafb41209b9e337a987f2887, State: Running, Role: LEADER
I20260809 22:53:54.935386 7644 consensus_queue.cc:260] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:54.945751 7644 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 3 FOLLOWER]: Advancing to term 4
I20260809 22:53:54.957779 7711 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:54.959720 7549 raft_consensus.cc:2379] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 493890a07f5f4d34b9f48a3da5421253 in current term 4: Already voted for candidate 055f6d77fb794a629017dd736c1a6060 in this term.
I20260809 22:53:54.962805 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253"
I20260809 22:53:54.966020 7711 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 4 election: Requested vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:54.958024 7853 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:54.968034 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887"
I20260809 22:53:54.972652 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887"
I20260809 22:53:54.975800 7463 raft_consensus.cc:2379] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 055f6d77fb794a629017dd736c1a6060 in current term 4: Already voted for candidate 493890a07f5f4d34b9f48a3da5421253 in this term.
I20260809 22:53:54.979889 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060"
I20260809 22:53:54.980895 7853 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 4 election: Requested vote from peers 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:54.982146 7549 raft_consensus.cc:2379] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 493890a07f5f4d34b9f48a3da5421253 in current term 4: Already voted for candidate 055f6d77fb794a629017dd736c1a6060 in this term.
I20260809 22:53:54.986948 7426 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493890a07f5f4d34b9f48a3da5421253; no voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887
I20260809 22:53:54.992884 7853 raft_consensus.cc:2726] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 22:53:55.040225 7853 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8db860deafb41209b9e337a987f2887)
I20260809 22:53:55.040933 7853 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:55.044214 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:55.044916 7645 raft_consensus.cc:2454] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493890a07f5f4d34b9f48a3da5421253 in term 4.
I20260809 22:53:55.044837 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:55.046262 7422 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493890a07f5f4d34b9f48a3da5421253, d8db860deafb41209b9e337a987f2887; no voters:
I20260809 22:53:55.046931 7853 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:55.047169 7837 raft_consensus.cc:2781] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 22:53:55.047587 7837 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8db860deafb41209b9e337a987f2887)
I20260809 22:53:55.047948 7837 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 3 FOLLOWER]: Advancing to term 4
I20260809 22:53:55.063107 7644 raft_consensus.cc:2454] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 055f6d77fb794a629017dd736c1a6060 in term 4.
I20260809 22:53:55.063124 7837 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:55.065575 7513 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters: 493890a07f5f4d34b9f48a3da5421253
I20260809 22:53:55.066326 7837 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 4 election: Requested vote from peers d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:55.067104 7711 raft_consensus.cc:2781] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 22:53:55.067644 7711 raft_consensus.cc:686] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 4 LEADER]: Becoming Leader. State: Replica: 055f6d77fb794a629017dd736c1a6060, State: Running, Role: LEADER
I20260809 22:53:55.068323 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060"
I20260809 22:53:55.068516 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887"
I20260809 22:53:55.068571 7711 consensus_queue.cc:237] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:55.074846 7711 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8db860deafb41209b9e337a987f2887)
I20260809 22:53:55.076215 7711 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:55.080276 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:55.082924 7711 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583), 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533)
I20260809 22:53:55.083626 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:55.081821 7153 catalog_manager.cc:5495] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 reported cstate change: term changed from 3 to 4, leader changed from d8db860deafb41209b9e337a987f2887 (127.5.105.131) to 055f6d77fb794a629017dd736c1a6060 (127.5.105.130). New cstate: current_term: 4 leader_uuid: "055f6d77fb794a629017dd736c1a6060" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 22:53:55.084513 7463 raft_consensus.cc:2379] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 055f6d77fb794a629017dd736c1a6060 in current term 4: Already voted for candidate 493890a07f5f4d34b9f48a3da5421253 in this term.
I20260809 22:53:55.085850 7513 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060; no voters: 493890a07f5f4d34b9f48a3da5421253, d8db860deafb41209b9e337a987f2887
I20260809 22:53:55.087059 7711 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 3 FOLLOWER]: Advancing to term 4
I20260809 22:53:55.160198 7153 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44858:
table { table_id: "842a76a4c8694fd187d3c04383286ae1" table_name: "test-table" } new_table_name: "NewTableName"
I20260809 22:53:55.167287 7644 raft_consensus.cc:2454] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 493890a07f5f4d34b9f48a3da5421253 in term 4.
I20260809 22:53:55.168761 7422 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 493890a07f5f4d34b9f48a3da5421253, d8db860deafb41209b9e337a987f2887; no voters: 055f6d77fb794a629017dd736c1a6060
I20260809 22:53:55.169682 7711 raft_consensus.cc:2726] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 22:53:55.173843 7837 raft_consensus.cc:2781] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 22:53:55.174724 7837 raft_consensus.cc:686] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 4 LEADER]: Becoming Leader. State: Replica: 493890a07f5f4d34b9f48a3da5421253, State: Running, Role: LEADER
I20260809 22:53:55.176023 7837 consensus_queue.cc:237] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:55.211923 7463 raft_consensus.cc:1261] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Refusing update from remote peer 055f6d77fb794a629017dd736c1a6060: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 22:53:55.212894 7644 raft_consensus.cc:1261] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Refusing update from remote peer 055f6d77fb794a629017dd736c1a6060: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 22:53:55.217293 7851 consensus_queue.cc:1035] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 22:53:55.209851 7157 catalog_manager.cc:5495] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 reported cstate change: term changed from 3 to 4, leader changed from d8db860deafb41209b9e337a987f2887 (127.5.105.131) to 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129). New cstate: current_term: 4 leader_uuid: "493890a07f5f4d34b9f48a3da5421253" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } health_report { overall_health: UNKNOWN } } }
I20260809 22:53:55.220417 7851 consensus_queue.cc:1035] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 22:53:55.261286 7853 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 055f6d77fb794a629017dd736c1a6060)
I20260809 22:53:55.261853 7853 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:53:55.297580 7160 catalog_manager.cc:5587] TS d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583) does not have the latest schema for tablet 2ad6955771cc4caa8eda877ded5a808a (table NewTableName [id=842a76a4c8694fd187d3c04383286ae1]). Expected version 1 got 0
I20260809 22:53:55.301467 7644 raft_consensus.cc:1261] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Refusing update from remote peer 493890a07f5f4d34b9f48a3da5421253: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260809 22:53:55.326288 7549 raft_consensus.cc:1261] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Refusing update from remote peer 493890a07f5f4d34b9f48a3da5421253: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260809 22:53:55.319187 7837 consensus_queue.cc:1035] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260809 22:53:55.338522 7837 consensus_queue.cc:1035] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [LEADER]: Connected to new peer: Peer: permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 22:53:55.361948 5542 master.cc:537] Master@127.5.105.190:34015 shutting down...
I20260809 22:53:55.386674 7157 catalog_manager.cc:5587] TS 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533) does not have the latest schema for tablet 2ad6955771cc4caa8eda877ded5a808a (table NewTableName [id=842a76a4c8694fd187d3c04383286ae1]). Expected version 1 got 0
I20260809 22:53:55.402526 7851 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:53:55.409085 7851 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:53:55.426898 7160 catalog_manager.cc:5587] TS 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821) does not have the latest schema for tablet 2ad6955771cc4caa8eda877ded5a808a (table NewTableName [id=842a76a4c8694fd187d3c04383286ae1]). Expected version 1 got 0
I20260809 22:53:55.432636 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
W20260809 22:53:55.436021 7630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47072: Illegal state: replica d8db860deafb41209b9e337a987f2887 is not leader of this config: current role FOLLOWER
W20260809 22:53:55.440404 7144 catalog_manager.cc:4920] TS d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583): alter failed for tablet 19ef6c21cc3c497483e1ddb9b9fe17ef (table NewTableName [id=842a76a4c8694fd187d3c04383286ae1]): Illegal state: replica d8db860deafb41209b9e337a987f2887 is not leader of this config: current role FOLLOWER
I20260809 22:53:55.430303 7853 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821), d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:55.443276 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:53:55.443194 7642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:53:55.447055 7513 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060; no voters: 493890a07f5f4d34b9f48a3da5421253, d8db860deafb41209b9e337a987f2887
I20260809 22:53:55.448208 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:53:55.452543 7426 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493890a07f5f4d34b9f48a3da5421253; no voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887
I20260809 22:53:55.442014 7851 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583), 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533)
I20260809 22:53:55.453975 7853 raft_consensus.cc:2726] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 22:53:55.499346 7711 raft_consensus.cc:2726] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 22:53:55.590513 7873 tablet.cc:1722] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260809 22:53:55.588706 7874 tablet.cc:1722] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260809 22:53:55.593374 7875 tablet.cc:1722] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260809 22:53:55.593475 7876 tablet.cc:1722] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260809 22:53:55.606767 7871 tablet.cc:1722] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260809 22:53:55.608353 7872 tablet.cc:1722] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260809 22:53:55.680325 7881 tablet.cc:1722] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260809 22:53:55.685565 7878 tablet.cc:1722] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260809 22:53:55.689847 7879 tablet.cc:1722] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260809 22:53:55.734316 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 3 LEADER]: Raft consensus shutting down.
I20260809 22:53:55.735827 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 22:53:55.738006 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: stopping tablet replica
I20260809 22:53:55.782646 5542 master.cc:552] Master@127.5.105.190:34015 shutdown complete.
I20260809 22:53:55.885531 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 23b696fe94a441ba93177a12ff465c26)
I20260809 22:53:55.885977 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:55.887652 7885 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015), 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689)
I20260809 22:53:55.889029 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "75d10a05561a482f9823bab385db5ed3" is_pre_election: true
W20260809 22:53:55.890165 7293 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.105.190:34015: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20260809 22:53:55.893193 7293 leader_election.cc:334] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015): Network error: Client connection negotiation failed: client connection to 127.5.105.190:34015: connect: Connection refused (error 111)
I20260809 22:53:55.893666 7293 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac; no voters: 23b696fe94a441ba93177a12ff465c26, 75d10a05561a482f9823bab385db5ed3
I20260809 22:53:55.894951 7885 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 22:53:55.919931 7888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 23b696fe94a441ba93177a12ff465c26)
I20260809 22:53:55.920547 7888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:55.922838 7888 leader_election.cc:288] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015), 1ed55e55e8fd4ad4830e8fd57d1d7cac (127.5.105.188:45511)
I20260809 22:53:55.924340 7323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75d10a05561a482f9823bab385db5ed3" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" is_pre_election: true
I20260809 22:53:55.924973 7323 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75d10a05561a482f9823bab385db5ed3 in term 3.
I20260809 22:53:55.926380 7218 leader_election.cc:302] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac, 75d10a05561a482f9823bab385db5ed3; no voters:
I20260809 22:53:55.927282 7888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 22:53:55.927671 7888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 23b696fe94a441ba93177a12ff465c26)
I20260809 22:53:55.928088 7888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 3 FOLLOWER]: Advancing to term 4
W20260809 22:53:55.928915 7219 leader_election.cc:334] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015): Network error: Client connection negotiation failed: client connection to 127.5.105.190:34015: connect: Connection refused (error 111)
I20260809 22:53:55.935669 7888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:55.937788 7888 leader_election.cc:288] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [CANDIDATE]: Term 4 election: Requested vote from peers 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015), 1ed55e55e8fd4ad4830e8fd57d1d7cac (127.5.105.188:45511)
I20260809 22:53:55.938870 7323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75d10a05561a482f9823bab385db5ed3" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac"
I20260809 22:53:55.939514 7323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 3 FOLLOWER]: Advancing to term 4
W20260809 22:53:55.942720 7219 leader_election.cc:334] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015): Network error: Client connection negotiation failed: client connection to 127.5.105.190:34015: connect: Connection refused (error 111)
I20260809 22:53:55.948791 7323 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75d10a05561a482f9823bab385db5ed3 in term 4.
I20260809 22:53:55.949992 7218 leader_election.cc:302] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac, 75d10a05561a482f9823bab385db5ed3; no voters: 23b696fe94a441ba93177a12ff465c26
I20260809 22:53:55.950884 7888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 22:53:55.952261 7888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 4 LEADER]: Becoming Leader. State: Replica: 75d10a05561a482f9823bab385db5ed3, State: Running, Role: LEADER
I20260809 22:53:55.953291 7888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:55.959347 7892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75d10a05561a482f9823bab385db5ed3. Latest consensus state: current_term: 4 leader_uuid: "75d10a05561a482f9823bab385db5ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:55.960258 7892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:55.961629 7893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:53:55.968380 7893 catalog_manager.cc:669] Loaded metadata for table NewTableName [id=842a76a4c8694fd187d3c04383286ae1]
I20260809 22:53:55.975740 7893 tablet_loader.cc:96] loaded metadata for tablet 119c8382d78c4413aefb2d2d9077d47e (table NewTableName [id=842a76a4c8694fd187d3c04383286ae1])
I20260809 22:53:55.977185 7893 tablet_loader.cc:96] loaded metadata for tablet 19ef6c21cc3c497483e1ddb9b9fe17ef (table NewTableName [id=842a76a4c8694fd187d3c04383286ae1])
I20260809 22:53:55.978216 7893 tablet_loader.cc:96] loaded metadata for tablet 2ad6955771cc4caa8eda877ded5a808a (table NewTableName [id=842a76a4c8694fd187d3c04383286ae1])
I20260809 22:53:55.979429 7893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:53:55.982715 7893 catalog_manager.cc:1259] Loaded cluster ID: f511035ecb4f4aaa9b8234496968ea8e
I20260809 22:53:55.983013 7893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:53:55.987051 7893 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:53:55.991175 7893 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3: Loaded TSK: 0
I20260809 22:53:55.992434 7893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:53:56.006784 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:53:56.015707 7894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:56.017079 7895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:53:56.021489 7897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:53:56.021572 5542 server_base.cc:1034] running on GCE node
I20260809 22:53:56.023283 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:53:56.023607 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:53:56.023818 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316036023805 us; error 0 us; skew 500 ppm
I20260809 22:53:56.024590 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:53:56.028203 5542 webserver.cc:460] Webserver started at http://127.5.105.190:46497/ using document root <none> and password file <none>
I20260809 22:53:56.028877 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:53:56.029114 5542 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260809 22:53:56.035051 5542 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
W20260809 22:53:56.037744 7219 consensus_peers.cc:487] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 -> Peer 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015): Couldn't send request to peer 23b696fe94a441ba93177a12ff465c26. Status: Network error: Client connection negotiation failed: client connection to 127.5.105.190:34015: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 22:53:56.039883 7902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:53:56.040886 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 22:53:56.041272 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-0-root
uuid: "23b696fe94a441ba93177a12ff465c26"
format_stamp: "Formatted at 2026-08-09 22:53:48 on dist-test-slave-gp6n"
I20260809 22:53:56.041867 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:53:56.068706 7323 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 4 FOLLOWER]: Refusing update from remote peer 75d10a05561a482f9823bab385db5ed3: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 4 index: 16. (index mismatch)
I20260809 22:53:56.071206 7892 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 22:53:56.072134 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:53:56.074250 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:53:56.076431 7885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75d10a05561a482f9823bab385db5ed3. Latest consensus state: current_term: 4 leader_uuid: "75d10a05561a482f9823bab385db5ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:56.077345 7885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:56.088363 7892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "75d10a05561a482f9823bab385db5ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:56.090199 7892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:56.096154 7885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "75d10a05561a482f9823bab385db5ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:56.097364 7885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:56.159843 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.190:34015
I20260809 22:53:56.160143 7960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.190:34015 every 8 connection(s)
I20260809 22:53:56.162477 7498 heartbeater.cc:503] Master 127.5.105.189:33689 was elected leader, sending a full tablet report...
I20260809 22:53:56.171605 7674 heartbeater.cc:503] Master 127.5.105.189:33689 was elected leader, sending a full tablet report...
I20260809 22:53:56.177798 7962 sys_catalog.cc:263] Verifying existing consensus state
I20260809 22:53:56.188246 7585 heartbeater.cc:503] Master 127.5.105.189:33689 was elected leader, sending a full tablet report...
I20260809 22:53:56.201738 7962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: Bootstrap starting.
I20260809 22:53:56.233289 7586 heartbeater.cc:347] Connected to a master server at 127.5.105.190:34015
I20260809 22:53:56.272861 7236 catalog_manager.cc:6040] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3: NewTableName [id=842a76a4c8694fd187d3c04383286ae1] alter complete (version 1)
I20260809 22:53:56.313374 7499 heartbeater.cc:347] Connected to a master server at 127.5.105.190:34015
I20260809 22:53:56.319861 7675 heartbeater.cc:347] Connected to a master server at 127.5.105.190:34015
I20260809 22:53:56.332172 7962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=7 ignored=0} mutations{seen=12 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 22:53:56.333621 7962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: Bootstrap complete.
I20260809 22:53:56.337193 7962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:56.338172 7962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b696fe94a441ba93177a12ff465c26, State: Initialized, Role: FOLLOWER
I20260809 22:53:56.339048 7962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:53:56.341280 7972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:56.342234 7972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:56.343140 7962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:53:56.345005 7933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 3 FOLLOWER]: Advancing to term 4
I20260809 22:53:56.415138 7982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "75d10a05561a482f9823bab385db5ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:56.416015 7982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:56.417286 7972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75d10a05561a482f9823bab385db5ed3. Latest consensus state: current_term: 4 leader_uuid: "75d10a05561a482f9823bab385db5ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:56.418073 7972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:53:56.426364 7989 catalog_manager.cc:1259] Loaded cluster ID: f511035ecb4f4aaa9b8234496968ea8e
I20260809 22:53:56.426733 7989 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: loading cluster ID for follower catalog manager: success
I20260809 22:53:56.432667 7989 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: acquiring CA information for follower catalog manager: success
I20260809 22:53:56.436969 7989 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 22:53:56.462918 7892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "75d10a05561a482f9823bab385db5ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:53:56.463876 7892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: This master's current role is: LEADER
I20260809 22:53:56.512261 7922 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d8db860deafb41209b9e337a987f2887" instance_seqno: 1786316029827912) as {username='slave'} at 127.0.0.1:45116; Asking this server to re-register.
I20260809 22:53:56.514158 7675 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:56.512259 7921 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "055f6d77fb794a629017dd736c1a6060" instance_seqno: 1786316029632769) as {username='slave'} at 127.0.0.1:45096; Asking this server to re-register.
I20260809 22:53:56.517505 7586 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:56.519454 7921 ts_manager.cc:194] Registered new tserver with Master: d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583)
I20260809 22:53:56.520988 7922 ts_manager.cc:194] Registered new tserver with Master: 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:53:56.528029 7922 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" instance_seqno: 1786316029469082) as {username='slave'} at 127.0.0.1:45132; Asking this server to re-register.
I20260809 22:53:56.529501 7499 heartbeater.cc:464] Registering TS with master...
I20260809 22:53:56.533202 7922 ts_manager.cc:194] Registered new tserver with Master: 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533)
I20260809 22:54:06.416054 8013 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 493890a07f5f4d34b9f48a3da5421253)
I20260809 22:54:06.417292 8013 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:54:06.421496 8013 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583), 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533)
I20260809 22:54:06.685122 8015 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 055f6d77fb794a629017dd736c1a6060)
I20260809 22:54:06.685882 8015 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:54:06.748960 8015 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:54:06.750797 8015 raft_consensus.cc:482] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 055f6d77fb794a629017dd736c1a6060)
I20260809 22:54:06.757284 8015 raft_consensus.cc:504] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:54:06.777208 8015 leader_election.cc:288] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:54:06.795154 8016 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 493890a07f5f4d34b9f48a3da5421253)
I20260809 22:54:06.795610 8014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 75d10a05561a482f9823bab385db5ed3)
I20260809 22:54:06.795919 8016 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:54:06.796430 8014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:54:06.798476 8016 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:54:06.799593 8014 leader_election.cc:288] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689), 1ed55e55e8fd4ad4830e8fd57d1d7cac (127.5.105.188:45511)
W20260809 22:54:06.999030 7602 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.5.105.129:33533 timed out after 0.200s (SENT) [suppressed 4 similar messages]
W20260809 22:54:07.397888 7904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.5.105.189:33689, user_credentials={real_user=slave}} blocked reactor thread for 315883us
W20260809 22:54:07.403678 7602 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.5.105.129:33533, user_credentials={real_user=slave}} blocked reactor thread for 404810us
W20260809 22:54:07.605697 7602 leader_election.cc:334] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533): Timed out: RequestConsensusVote RPC to 127.5.105.129:33533 timed out after 0.200s (SENT)
W20260809 22:54:07.606734 7603 leader_election.cc:334] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821): Timed out: RequestConsensusVote RPC to 127.5.105.130:33821 timed out after 0.200s (SENT)
I20260809 22:54:07.607262 7603 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8db860deafb41209b9e337a987f2887; no voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253
W20260809 22:54:07.682166 7904 leader_election.cc:334] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689): Timed out: connection negotiation to 127.5.105.189:33689 for RPC RequestConsensusVote timed out after 0.282s (ON_OUTBOUND_QUEUE)
W20260809 22:54:07.687378 7904 leader_election.cc:334] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1ed55e55e8fd4ad4830e8fd57d1d7cac (127.5.105.188:45511): Timed out: connection negotiation to 127.5.105.188:45511 for RPC RequestConsensusVote timed out after 0.282s (ON_OUTBOUND_QUEUE)
I20260809 22:54:07.687938 7904 leader_election.cc:302] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23b696fe94a441ba93177a12ff465c26; no voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac, 75d10a05561a482f9823bab385db5ed3
I20260809 22:54:07.841271 8021 raft_consensus.cc:2726] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 22:54:07.842849 7603 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.5.105.130:33821, user_credentials={real_user=slave}} blocked reactor thread for 236735us
W20260809 22:54:07.875808 7603 leader_election.cc:334] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821): Timed out: RequestConsensusVote RPC to 127.5.105.130:33821 timed out after 0.256s (SENT)
W20260809 22:54:07.876710 7603 leader_election.cc:334] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821): Timed out: RequestConsensusVote RPC to 127.5.105.130:33821 timed out after 0.298s (SENT)
W20260809 22:54:07.884045 7602 leader_election.cc:334] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533): Timed out: RequestConsensusVote RPC to 127.5.105.129:33533 timed out after 0.256s (SENT)
I20260809 22:54:07.884793 7602 leader_election.cc:302] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8db860deafb41209b9e337a987f2887; no voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253
W20260809 22:54:07.886485 7602 leader_election.cc:334] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533): Timed out: RequestConsensusVote RPC to 127.5.105.129:33533 timed out after 0.298s (SENT)
I20260809 22:54:07.887084 7602 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8db860deafb41209b9e337a987f2887; no voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253
W20260809 22:54:07.897910 7904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.5.105.188:45511, user_credentials={real_user=slave}} blocked reactor thread for 210575us
I20260809 22:54:07.897910 8022 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 22:54:07.994179 8021 raft_consensus.cc:2726] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 22:54:08.022708 8021 raft_consensus.cc:2726] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 22:54:08.197247 7602 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.5.105.129:33533, user_credentials={real_user=slave}} blocked reactor thread for 310761us
I20260809 22:54:08.216645 8023 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 493890a07f5f4d34b9f48a3da5421253)
I20260809 22:54:08.217324 8023 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:54:08.219997 8023 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:54:08.229226 7236 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50914 (request call id 151) took 1428 ms. Trace:
I20260809 22:54:08.229816 7236 rpcz_store.cc:276] 0809 22:54:06.801079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 22:54:08.227998 (+1426919us) service_pool.cc:227] Handling call
0809 22:54:08.229057 (+ 1059us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":0}
I20260809 22:54:08.230695 8022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 75d10a05561a482f9823bab385db5ed3)
I20260809 22:54:08.231345 8022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:54:08.233685 8022 leader_election.cc:288] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689), 1ed55e55e8fd4ad4830e8fd57d1d7cac (127.5.105.188:45511)
I20260809 22:54:08.243049 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:54:08.243884 7644 raft_consensus.cc:2454] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 055f6d77fb794a629017dd736c1a6060 in term 4.
I20260809 22:54:08.242882 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:54:08.250727 7513 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters:
I20260809 22:54:08.253093 8025 raft_consensus.cc:2781] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 22:54:08.253628 8025 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 493890a07f5f4d34b9f48a3da5421253)
I20260809 22:54:08.254108 8025 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Advancing to term 5
I20260809 22:54:08.266938 8025 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:54:08.270282 7644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887"
I20260809 22:54:08.270675 8025 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 5 election: Requested vote from peers d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583), 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533)
I20260809 22:54:08.271098 7644 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Advancing to term 5
I20260809 22:54:08.274894 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "055f6d77fb794a629017dd736c1a6060" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253"
I20260809 22:54:08.281596 8033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 75d10a05561a482f9823bab385db5ed3)
I20260809 22:54:08.283857 7323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b696fe94a441ba93177a12ff465c26" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" is_pre_election: true
I20260809 22:54:08.284919 7644 raft_consensus.cc:2454] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 055f6d77fb794a629017dd736c1a6060 in term 5.
I20260809 22:54:08.285856 7642 raft_consensus.cc:1226] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 5 FOLLOWER]: Rejecting Update request from peer 493890a07f5f4d34b9f48a3da5421253 for earlier term 4. Current term is 5. Ops: []
I20260809 22:54:08.286128 7513 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887; no voters: 493890a07f5f4d34b9f48a3da5421253
I20260809 22:54:08.282327 8033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:54:08.287153 8025 raft_consensus.cc:2781] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 22:54:08.287897 7323 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b696fe94a441ba93177a12ff465c26 in term 4.
I20260809 22:54:08.289621 8032 consensus_queue.cc:1046] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 22:54:08.292451 8033 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015), 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689)
I20260809 22:54:08.293608 7904 leader_election.cc:302] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac, 23b696fe94a441ba93177a12ff465c26; no voters:
I20260809 22:54:08.294899 7247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "75d10a05561a482f9823bab385db5ed3" is_pre_election: true
I20260809 22:54:08.294940 8036 raft_consensus.cc:3032] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 22:54:08.295697 8036 raft_consensus.cc:729] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 493890a07f5f4d34b9f48a3da5421253, State: Running, Role: LEADER
I20260809 22:54:08.297039 8022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 22:54:08.297737 8022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 75d10a05561a482f9823bab385db5ed3)
I20260809 22:54:08.298199 8022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 4 FOLLOWER]: Advancing to term 5
I20260809 22:54:08.297073 8036 consensus_queue.cc:260] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:54:08.311849 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:54:08.313879 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:54:08.316093 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b696fe94a441ba93177a12ff465c26" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "75d10a05561a482f9823bab385db5ed3" is_pre_election: true
I20260809 22:54:08.319067 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:54:08.327773 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:54:08.328208 8025 raft_consensus.cc:686] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 5 LEADER]: Becoming Leader. State: Replica: 055f6d77fb794a629017dd736c1a6060, State: Running, Role: LEADER
I20260809 22:54:08.329039 8025 consensus_queue.cc:237] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:54:08.329766 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:54:08.330363 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:54:08.331760 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:54:08.332058 7464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:54:08.332832 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:54:08.332930 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:54:08.336115 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:54:08.336824 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:54:08.337983 7549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:54:08.340523 7932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "23b696fe94a441ba93177a12ff465c26" is_pre_election: true
I20260809 22:54:08.341496 7603 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8db860deafb41209b9e337a987f2887; no voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253
I20260809 22:54:08.340469 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:54:08.342818 8023 raft_consensus.cc:2726] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 22:54:08.346967 7293 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac; no voters: 23b696fe94a441ba93177a12ff465c26, 75d10a05561a482f9823bab385db5ed3
I20260809 22:54:08.352572 8022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:54:08.353988 7933 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 5 FOLLOWER]: Rejecting Update request from peer 75d10a05561a482f9823bab385db5ed3 for earlier term 4. Current term is 5. Ops: []
I20260809 22:54:08.355736 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b696fe94a441ba93177a12ff465c26" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "75d10a05561a482f9823bab385db5ed3"
I20260809 22:54:08.357120 7323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b696fe94a441ba93177a12ff465c26" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac"
I20260809 22:54:08.358605 8022 leader_election.cc:288] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 5 election: Requested vote from peers 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689), 1ed55e55e8fd4ad4830e8fd57d1d7cac (127.5.105.188:45511)
I20260809 22:54:08.358918 7904 leader_election.cc:302] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23b696fe94a441ba93177a12ff465c26; no voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac, 75d10a05561a482f9823bab385db5ed3
I20260809 22:54:08.359899 8022 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 22:54:08.363500 8033 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 22:54:09.152460 7218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.105.188:45511, user_credentials={real_user=slave}} blocked reactor thread for 76388.9us
W20260809 22:54:09.272660 7553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60486 (request call id 69) took 963 ms (client timeout 200 ms). Trace:
W20260809 22:54:09.273417 7553 rpcz_store.cc:269] 0809 22:54:08.309470 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 22:54:08.313394 (+ 3924us) service_pool.cc:227] Handling call
0809 22:54:09.272345 (+958951us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b5400340154. stack: 00007fe3cab13033 00007fe3cab12f30 00007fe3ca471534 00000000003b4464 00000000003b43f9 00007fe3d1337471 00007fe3d13b62ae 00007fe3d35bfc27 00007fe3d0f47c04 00007fe3d0f47b92 00007fe3d0f47ac1 00007fe3d0f47a3c 00007fe3d0f46cf2 00007fe3cf67394c 00007fe3cf672d86 00007fe3cf67272f
0809 22:54:09.272552 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":60042496}
I20260809 22:54:09.284261 8045 raft_consensus.cc:482] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 055f6d77fb794a629017dd736c1a6060)
I20260809 22:54:09.283957 8044 raft_consensus.cc:482] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 055f6d77fb794a629017dd736c1a6060)
I20260809 22:54:09.287104 8045 raft_consensus.cc:504] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:54:09.287276 8044 raft_consensus.cc:504] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } } peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } }
I20260809 22:54:09.290436 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:54:09.292858 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:54:09.293982 8043 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:54:09.294633 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119c8382d78c4413aefb2d2d9077d47e" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:54:09.291627 8045 leader_election.cc:288] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:54:09.294545 8043 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:54:09.296025 7645 raft_consensus.cc:1261] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 5 FOLLOWER]: Refusing update from remote peer 055f6d77fb794a629017dd736c1a6060: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 22:54:09.296558 7602 leader_election.cc:302] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8db860deafb41209b9e337a987f2887; no voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253
I20260809 22:54:09.296773 8036 raft_consensus.cc:3037] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Advancing to term 5
I20260809 22:54:09.297392 8045 raft_consensus.cc:2726] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 22:54:09.298178 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:54:09.298735 8044 leader_election.cc:288] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:54:09.299115 8034 consensus_queue.cc:1035] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 22:54:09.301479 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6955771cc4caa8eda877ded5a808a" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:54:09.300640 8043 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:54:09.300004 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "d8db860deafb41209b9e337a987f2887" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "493890a07f5f4d34b9f48a3da5421253" is_pre_election: true
I20260809 22:54:09.304253 7603 leader_election.cc:302] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8db860deafb41209b9e337a987f2887; no voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253
I20260809 22:54:09.305078 7602 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8db860deafb41209b9e337a987f2887; no voters: 055f6d77fb794a629017dd736c1a6060, 493890a07f5f4d34b9f48a3da5421253
I20260809 22:54:09.305454 8045 raft_consensus.cc:2726] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 22:54:09.308840 8047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:54:09.309593 8047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:54:09.310571 8040 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 }, Status: INVALID_TERM, Last received: 4.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 22:54:09.311792 8040 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 22:54:09.312407 8038 raft_consensus.cc:482] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:54:09.313184 8045 raft_consensus.cc:2726] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 22:54:09.312861 8038 raft_consensus.cc:504] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } }
I20260809 22:54:09.314599 7465 raft_consensus.cc:2454] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8db860deafb41209b9e337a987f2887 in term 5.
I20260809 22:54:09.316445 8047 leader_election.cc:288] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689), 1ed55e55e8fd4ad4830e8fd57d1d7cac (127.5.105.188:45511)
I20260809 22:54:09.318670 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "055f6d77fb794a629017dd736c1a6060" is_pre_election: true
I20260809 22:54:09.319769 7645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ef6c21cc3c497483e1ddb9b9fe17ef" candidate_uuid: "493890a07f5f4d34b9f48a3da5421253" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8db860deafb41209b9e337a987f2887" is_pre_election: true
I20260809 22:54:09.320457 8038 leader_election.cc:288] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8db860deafb41209b9e337a987f2887 (127.5.105.131:33583), 055f6d77fb794a629017dd736c1a6060 (127.5.105.130:33821)
I20260809 22:54:09.321993 7426 leader_election.cc:302] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 493890a07f5f4d34b9f48a3da5421253; no voters: 055f6d77fb794a629017dd736c1a6060, d8db860deafb41209b9e337a987f2887
I20260809 22:54:09.325816 8038 raft_consensus.cc:2726] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 22:54:09.325016 7933 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 5 FOLLOWER]: Rejecting Update request from peer 75d10a05561a482f9823bab385db5ed3 for earlier term 4. Current term is 5. Ops: []
I20260809 22:54:09.312121 8040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 75d10a05561a482f9823bab385db5ed3, State: Running, Role: LEADER
I20260809 22:54:09.327979 8040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 4.17, Last appended by leader: 17, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:54:09.329342 8040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 4 FOLLOWER]: Advancing to term 5
W20260809 22:54:09.342147 7465 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39140 (request call id 75) took 1011 ms (client timeout 200 ms). Trace:
I20260809 22:54:09.342516 7466 raft_consensus.cc:1261] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 5 FOLLOWER]: Refusing update from remote peer 055f6d77fb794a629017dd736c1a6060: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260809 22:54:09.342667 7465 rpcz_store.cc:269] 0809 22:54:08.330603 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 22:54:08.331328 (+ 725us) service_pool.cc:227] Handling call
0809 22:54:09.341911 (+1010583us) spinlock_profiling.cc:243] Waited 982 ms on lock 0x7b54003502d4. stack: 00007fe3cab13033 00007fe3cab12f30 00007fe3ca471534 00000000003b4464 00000000003b43f9 00007fe3d1337471 00007fe3d13b62ae 00007fe3d35bfc27 00007fe3d0f47c04 00007fe3d0f47b92 00007fe3d0f47ac1 00007fe3d0f47a3c 00007fe3d0f46cf2 00007fe3cf67394c 00007fe3cf672d86 00007fe3cf67272f
0809 22:54:09.342041 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2258502400}
I20260809 22:54:09.350595 8034 consensus_queue.cc:1035] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 22:54:09.355834 7323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b696fe94a441ba93177a12ff465c26" candidate_term: 6 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" is_pre_election: true
I20260809 22:54:09.363770 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b696fe94a441ba93177a12ff465c26" candidate_term: 6 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "75d10a05561a482f9823bab385db5ed3" is_pre_election: true
I20260809 22:54:09.364434 7248 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b696fe94a441ba93177a12ff465c26 in term 5.
I20260809 22:54:09.368510 7904 leader_election.cc:302] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b696fe94a441ba93177a12ff465c26, 75d10a05561a482f9823bab385db5ed3; no voters:
I20260809 22:54:09.369946 8047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 22:54:09.370299 8047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:54:09.370805 8047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 5 FOLLOWER]: Advancing to term 6
I20260809 22:54:09.379798 8047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:54:09.381734 8047 leader_election.cc:288] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 6 election: Requested vote from peers 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689), 1ed55e55e8fd4ad4830e8fd57d1d7cac (127.5.105.188:45511)
I20260809 22:54:09.382440 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b696fe94a441ba93177a12ff465c26" candidate_term: 6 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "75d10a05561a482f9823bab385db5ed3"
I20260809 22:54:09.383116 7248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 5 FOLLOWER]: Advancing to term 6
I20260809 22:54:09.383124 7323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b696fe94a441ba93177a12ff465c26" candidate_term: 6 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac"
I20260809 22:54:09.390700 7248 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b696fe94a441ba93177a12ff465c26 in term 6.
I20260809 22:54:09.391816 7904 leader_election.cc:302] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23b696fe94a441ba93177a12ff465c26, 75d10a05561a482f9823bab385db5ed3; no voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac
I20260809 22:54:09.392536 8047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 22:54:09.393909 8047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 6 LEADER]: Becoming Leader. State: Replica: 23b696fe94a441ba93177a12ff465c26, State: Running, Role: LEADER
I20260809 22:54:09.394778 8047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 4.17, Last appended by leader: 17, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:54:09.400218 8053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b696fe94a441ba93177a12ff465c26. Latest consensus state: current_term: 6 leader_uuid: "23b696fe94a441ba93177a12ff465c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:54:09.401129 8053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:09.403251 8055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:54:09.412576 8055 catalog_manager.cc:669] Loaded metadata for table NewTableName [id=842a76a4c8694fd187d3c04383286ae1]
I20260809 22:54:09.420534 8055 tablet_loader.cc:96] loaded metadata for tablet 119c8382d78c4413aefb2d2d9077d47e (table NewTableName [id=842a76a4c8694fd187d3c04383286ae1])
I20260809 22:54:09.422078 8055 tablet_loader.cc:96] loaded metadata for tablet 19ef6c21cc3c497483e1ddb9b9fe17ef (table NewTableName [id=842a76a4c8694fd187d3c04383286ae1])
I20260809 22:54:09.423596 8055 tablet_loader.cc:96] loaded metadata for tablet 2ad6955771cc4caa8eda877ded5a808a (table NewTableName [id=842a76a4c8694fd187d3c04383286ae1])
I20260809 22:54:09.425047 8055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:54:09.430271 8055 catalog_manager.cc:1259] Loaded cluster ID: f511035ecb4f4aaa9b8234496968ea8e
I20260809 22:54:09.430589 8055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:54:09.436866 8055 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:54:09.442099 8055 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26: Loaded TSK: 0
I20260809 22:54:09.443603 8055 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 22:54:09.459151 7579 debug-util.cc:398] Leaking SignalData structure 0x7b08001d8060 after lost signal to thread 5546
I20260809 22:54:09.459344 7922 catalog_manager.cc:5495] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 reported cstate change: term changed from 4 to 5, leader changed from 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129) to 055f6d77fb794a629017dd736c1a6060 (127.5.105.130). New cstate: current_term: 5 leader_uuid: "055f6d77fb794a629017dd736c1a6060" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8db860deafb41209b9e337a987f2887" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "493890a07f5f4d34b9f48a3da5421253" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 33533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "055f6d77fb794a629017dd736c1a6060" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 33821 } health_report { overall_health: HEALTHY } } }
W20260809 22:54:09.460330 7579 debug-util.cc:398] Leaking SignalData structure 0x7b08001d87c0 after lost signal to thread 7275
W20260809 22:54:09.461160 7579 debug-util.cc:398] Leaking SignalData structure 0x7b08001d8e80 after lost signal to thread 7350
W20260809 22:54:09.462231 7579 debug-util.cc:398] Leaking SignalData structure 0x7b08001de4e0 after lost signal to thread 7496
W20260809 22:54:09.463088 7579 debug-util.cc:398] Leaking SignalData structure 0x7b0800106360 after lost signal to thread 7582
W20260809 22:54:09.463944 7579 debug-util.cc:398] Leaking SignalData structure 0x7b080011bfa0 after lost signal to thread 7671
W20260809 22:54:09.464809 7579 debug-util.cc:398] Leaking SignalData structure 0x7b08000fc040 after lost signal to thread 7960
I20260809 22:54:09.469417 7323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 4 FOLLOWER]: Advancing to term 6
I20260809 22:54:09.477486 7323 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 6 FOLLOWER]: Refusing update from remote peer 23b696fe94a441ba93177a12ff465c26: Log matching property violated. Preceding OpId in replica: term: 4 index: 17. Preceding OpId from leader: term: 6 index: 19. (index mismatch)
I20260809 22:54:09.479104 8053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 22:54:09.484380 7248 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 6 FOLLOWER]: Refusing update from remote peer 23b696fe94a441ba93177a12ff465c26: Log matching property violated. Preceding OpId in replica: term: 4 index: 17. Preceding OpId from leader: term: 6 index: 19. (index mismatch)
I20260809 22:54:09.488579 8053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 22:54:09.525774 8061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 23b696fe94a441ba93177a12ff465c26)
I20260809 22:54:09.525501 8060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b696fe94a441ba93177a12ff465c26. Latest consensus state: current_term: 6 leader_uuid: "23b696fe94a441ba93177a12ff465c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:54:09.526434 8060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:54:09.526357 8061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } }
I20260809 22:54:09.531002 8061 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 23b696fe94a441ba93177a12ff465c26 (127.5.105.190:34015), 75d10a05561a482f9823bab385db5ed3 (127.5.105.189:33689)
I20260809 22:54:09.533577 7247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" candidate_term: 7 candidate_status { last_received { term: 6 index: 19 } } ignore_live_leader: false dest_uuid: "75d10a05561a482f9823bab385db5ed3" is_pre_election: true
I20260809 22:54:09.534797 8047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "23b696fe94a441ba93177a12ff465c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:54:09.535333 7933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" candidate_term: 7 candidate_status { last_received { term: 6 index: 19 } } ignore_live_leader: false dest_uuid: "23b696fe94a441ba93177a12ff465c26" is_pre_election: true
I20260809 22:54:09.535557 8047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:09.538470 7293 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ed55e55e8fd4ad4830e8fd57d1d7cac; no voters: 23b696fe94a441ba93177a12ff465c26, 75d10a05561a482f9823bab385db5ed3
I20260809 22:54:09.540611 8040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b696fe94a441ba93177a12ff465c26. Latest consensus state: current_term: 6 leader_uuid: "23b696fe94a441ba93177a12ff465c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:54:09.541601 8040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:54:09.545782 8047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "23b696fe94a441ba93177a12ff465c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:54:09.546581 8047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:09.557471 8061 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 22:54:09.560863 8060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "23b696fe94a441ba93177a12ff465c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:54:09.561882 8060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:54:09.564101 7675 heartbeater.cc:503] Master 127.5.105.190:34015 was elected leader, sending a full tablet report...
I20260809 22:54:09.580744 8040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "23b696fe94a441ba93177a12ff465c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b696fe94a441ba93177a12ff465c26" member_type: VOTER last_known_addr { host: "127.5.105.190" port: 34015 } } peers { permanent_uuid: "75d10a05561a482f9823bab385db5ed3" member_type: VOTER last_known_addr { host: "127.5.105.189" port: 33689 } } peers { permanent_uuid: "1ed55e55e8fd4ad4830e8fd57d1d7cac" member_type: VOTER last_known_addr { host: "127.5.105.188" port: 45511 } } }
I20260809 22:54:09.581620 8040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:54:09.588614 7499 heartbeater.cc:503] Master 127.5.105.190:34015 was elected leader, sending a full tablet report...
I20260809 22:54:09.654554 7586 heartbeater.cc:503] Master 127.5.105.190:34015 was elected leader, sending a full tablet report...
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:783: Failure
Expected equality of these values:
live_row_count
Which is: 802
table_info->GetMetrics()->live_row_count->value()
Which is: 0
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_util.cc:395: Failure
Failed
Timed out waiting for assertion to pass.
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:767: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
Actual: it does.
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:785: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 782, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
Actual: it does.
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:895: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20260809 22:54:26.503136 5542 tablet_server.cc:178] TabletServer@127.5.105.129:0 shutting down...
W20260809 22:54:26.531446 7513 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 7 similar messages]
W20260809 22:54:26.536810 7513 consensus_peers.cc:487] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 -> Peer 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533): Couldn't send request to peer 493890a07f5f4d34b9f48a3da5421253. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 22:54:26.542392 7513 consensus_peers.cc:487] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 -> Peer 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533): Couldn't send request to peer 493890a07f5f4d34b9f48a3da5421253. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 22:54:26.549705 7513 consensus_peers.cc:487] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 -> Peer 493890a07f5f4d34b9f48a3da5421253 (127.5.105.129:33533): Couldn't send request to peer 493890a07f5f4d34b9f48a3da5421253. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 22:54:26.554339 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:26.555323 5542 tablet_replica.cc:330] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253: stopping tablet replica
I20260809 22:54:26.556188 5542 raft_consensus.cc:2229] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:26.556823 5542 raft_consensus.cc:2258] T 119c8382d78c4413aefb2d2d9077d47e P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:26.559823 5542 tablet_replica.cc:330] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253: stopping tablet replica
I20260809 22:54:26.560376 5542 raft_consensus.cc:2229] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:26.560866 5542 raft_consensus.cc:2258] T 2ad6955771cc4caa8eda877ded5a808a P 493890a07f5f4d34b9f48a3da5421253 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:26.563484 5542 tablet_replica.cc:330] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253: stopping tablet replica
I20260809 22:54:26.563978 5542 raft_consensus.cc:2229] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:26.564479 5542 raft_consensus.cc:2258] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 493890a07f5f4d34b9f48a3da5421253 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:26.619091 5542 tablet_server.cc:194] TabletServer@127.5.105.129:0 shutdown complete.
I20260809 22:54:26.636838 5542 tablet_server.cc:178] TabletServer@127.5.105.130:0 shutting down...
I20260809 22:54:26.696091 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:26.696820 5542 tablet_replica.cc:330] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060: stopping tablet replica
I20260809 22:54:26.697517 5542 raft_consensus.cc:2229] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 4 LEADER]: Raft consensus shutting down.
I20260809 22:54:26.698377 5542 raft_consensus.cc:2258] T 119c8382d78c4413aefb2d2d9077d47e P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:26.701002 5542 tablet_replica.cc:330] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060: stopping tablet replica
I20260809 22:54:26.701535 5542 raft_consensus.cc:2229] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 5 LEADER]: Raft consensus shutting down.
I20260809 22:54:26.702391 5542 raft_consensus.cc:2258] T 19ef6c21cc3c497483e1ddb9b9fe17ef P 055f6d77fb794a629017dd736c1a6060 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:26.704975 5542 tablet_replica.cc:330] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060: stopping tablet replica
I20260809 22:54:26.705426 5542 raft_consensus.cc:2229] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 4 LEADER]: Raft consensus shutting down.
I20260809 22:54:26.706228 5542 raft_consensus.cc:2258] T 2ad6955771cc4caa8eda877ded5a808a P 055f6d77fb794a629017dd736c1a6060 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:26.751497 5542 tablet_server.cc:194] TabletServer@127.5.105.130:0 shutdown complete.
I20260809 22:54:26.766191 5542 tablet_server.cc:178] TabletServer@127.5.105.131:0 shutting down...
I20260809 22:54:26.801757 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:26.802569 5542 tablet_replica.cc:330] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887: stopping tablet replica
I20260809 22:54:26.803526 5542 raft_consensus.cc:2229] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:26.804277 5542 raft_consensus.cc:2258] T 119c8382d78c4413aefb2d2d9077d47e P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:26.807248 5542 tablet_replica.cc:330] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887: stopping tablet replica
I20260809 22:54:26.807716 5542 raft_consensus.cc:2229] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:26.808154 5542 raft_consensus.cc:2258] T 2ad6955771cc4caa8eda877ded5a808a P d8db860deafb41209b9e337a987f2887 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:26.810691 5542 tablet_replica.cc:330] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887: stopping tablet replica
I20260809 22:54:26.811141 5542 raft_consensus.cc:2229] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:26.811599 5542 raft_consensus.cc:2258] T 19ef6c21cc3c497483e1ddb9b9fe17ef P d8db860deafb41209b9e337a987f2887 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:26.833940 5542 tablet_server.cc:194] TabletServer@127.5.105.131:0 shutdown complete.
I20260809 22:54:26.848690 5542 master.cc:537] Master@127.5.105.190:34015 shutting down...
I20260809 22:54:26.877602 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 6 LEADER]: Raft consensus shutting down.
I20260809 22:54:26.878590 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 23b696fe94a441ba93177a12ff465c26 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:26.878950 5542 tablet_replica.cc:330] stopping tablet replica
I20260809 22:54:26.897764 5542 master.cc:552] Master@127.5.105.190:34015 shutdown complete.
I20260809 22:54:26.918578 5542 master.cc:537] Master@127.5.105.189:33689 shutting down...
I20260809 22:54:26.941191 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:26.941897 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:26.942350 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P 75d10a05561a482f9823bab385db5ed3: stopping tablet replica
I20260809 22:54:26.960989 5542 master.cc:552] Master@127.5.105.189:33689 shutdown complete.
I20260809 22:54:26.981431 5542 master.cc:537] Master@127.5.105.188:45511 shutting down...
I20260809 22:54:27.006916 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 6 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:27.007675 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac [term 6 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:27.008005 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1ed55e55e8fd4ad4830e8fd57d1d7cac: stopping tablet replica
I20260809 22:54:27.027196 5542 master.cc:552] Master@127.5.105.188:45511 shutdown complete.
I20260809 22:54:27.054700 5542 test_util.cc:175] -----------------------------------------------
I20260809 22:54:27.054948 5542 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786316013627029-5542-0
[ FAILED ] TsTabletManagerITest.TestTableStats (38163 ms)
[ RUN ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20260809 22:54:27.061277 5542 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.5.105.190:39521
I20260809 22:54:27.062393 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:27.068866 8102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:27.068900 8101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:27.070322 8104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:27.073186 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:27.074057 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:27.074249 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:27.074401 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316067074376 us; error 0 us; skew 500 ppm
I20260809 22:54:27.074951 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:27.077364 5542 webserver.cc:460] Webserver started at http://127.5.105.190:33097/ using document root <none> and password file <none>
I20260809 22:54:27.077834 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:27.078012 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:27.078295 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:27.079934 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/master-0-root/instance:
uuid: "4a4ee12ee8e74c5199546233ebe6d774"
format_stamp: "Formatted at 2026-08-09 22:54:27 on dist-test-slave-gp6n"
I20260809 22:54:27.084538 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 22:54:27.087823 8109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:27.088529 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 22:54:27.088819 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/master-0-root
uuid: "4a4ee12ee8e74c5199546233ebe6d774"
format_stamp: "Formatted at 2026-08-09 22:54:27 on dist-test-slave-gp6n"
I20260809 22:54:27.089090 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:27.101478 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:27.102777 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:27.144337 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.190:39521
I20260809 22:54:27.144436 8166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.190:39521 every 8 connection(s)
I20260809 22:54:27.148308 8167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:27.158720 8167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774: Bootstrap starting.
I20260809 22:54:27.163111 8167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:27.167289 8167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774: No bootstrap required, opened a new log
I20260809 22:54:27.169370 8167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4ee12ee8e74c5199546233ebe6d774" member_type: VOTER }
I20260809 22:54:27.169798 8167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:27.170013 8167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4ee12ee8e74c5199546233ebe6d774, State: Initialized, Role: FOLLOWER
I20260809 22:54:27.170588 8167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4ee12ee8e74c5199546233ebe6d774" member_type: VOTER }
I20260809 22:54:27.171054 8167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:54:27.171280 8167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:54:27.171550 8167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:27.175809 8167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4ee12ee8e74c5199546233ebe6d774" member_type: VOTER }
I20260809 22:54:27.176283 8167 leader_election.cc:302] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a4ee12ee8e74c5199546233ebe6d774; no voters:
I20260809 22:54:27.177424 8167 leader_election.cc:288] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:54:27.177881 8170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:27.179354 8170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [term 1 LEADER]: Becoming Leader. State: Replica: 4a4ee12ee8e74c5199546233ebe6d774, State: Running, Role: LEADER
I20260809 22:54:27.180027 8170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4ee12ee8e74c5199546233ebe6d774" member_type: VOTER }
I20260809 22:54:27.180337 8167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:54:27.184710 8171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a4ee12ee8e74c5199546233ebe6d774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4ee12ee8e74c5199546233ebe6d774" member_type: VOTER } }
I20260809 22:54:27.185336 8171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:27.186690 8173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a4ee12ee8e74c5199546233ebe6d774. Latest consensus state: current_term: 1 leader_uuid: "4a4ee12ee8e74c5199546233ebe6d774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a4ee12ee8e74c5199546233ebe6d774" member_type: VOTER } }
I20260809 22:54:27.187166 8173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:27.188625 8178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:54:27.196626 8178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:54:27.197764 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 22:54:27.205457 8178 catalog_manager.cc:1347] Generated new cluster ID: f422945e365243bb90478adee4be05c5
I20260809 22:54:27.205708 8178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:54:27.223703 8178 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:54:27.225291 8178 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:54:27.244457 8178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774: Generated new TSK 0
I20260809 22:54:27.245157 8178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:54:27.264520 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:27.271039 8191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:27.271562 8192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:27.274011 8195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:27.274276 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:27.275482 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:27.275658 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:27.275770 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316067275762 us; error 0 us; skew 500 ppm
I20260809 22:54:27.276219 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:27.278530 5542 webserver.cc:460] Webserver started at http://127.5.105.129:45499/ using document root <none> and password file <none>
I20260809 22:54:27.278959 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:27.279101 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:27.279301 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:27.280390 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-0-root/instance:
uuid: "ada126cc77e443318e820939d4e82587"
format_stamp: "Formatted at 2026-08-09 22:54:27 on dist-test-slave-gp6n"
I20260809 22:54:27.285320 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 22:54:27.288635 8201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:27.289402 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 22:54:27.289700 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "ada126cc77e443318e820939d4e82587"
format_stamp: "Formatted at 2026-08-09 22:54:27 on dist-test-slave-gp6n"
I20260809 22:54:27.290017 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:27.309942 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:27.311257 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:27.312775 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:54:27.315062 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:27.315267 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:27.315506 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:27.315685 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:27.357750 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:44943
I20260809 22:54:27.357872 8271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:44943 every 8 connection(s)
I20260809 22:54:27.362501 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:27.370271 8276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:27.370950 8277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:27.373991 5542 server_base.cc:1034] running on GCE node
W20260809 22:54:27.374992 8280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:27.376008 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:27.376281 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:27.376482 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316067376465 us; error 0 us; skew 500 ppm
I20260809 22:54:27.377185 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:27.378521 8272 heartbeater.cc:347] Connected to a master server at 127.5.105.190:39521
I20260809 22:54:27.378949 8272 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:27.379685 8272 heartbeater.cc:511] Master 127.5.105.190:39521 requested a full tablet report, sending...
I20260809 22:54:27.379930 5542 webserver.cc:460] Webserver started at http://127.5.105.130:43293/ using document root <none> and password file <none>
I20260809 22:54:27.380548 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:27.380784 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:27.381127 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:27.381953 8129 ts_manager.cc:194] Registered new tserver with Master: ada126cc77e443318e820939d4e82587 (127.5.105.129:44943)
I20260809 22:54:27.382369 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-1-root/instance:
uuid: "de2c9d4c5894484e95281c4b60fa46ef"
format_stamp: "Formatted at 2026-08-09 22:54:27 on dist-test-slave-gp6n"
I20260809 22:54:27.383940 8129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55472
I20260809 22:54:27.387373 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260809 22:54:27.390939 8285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:27.391872 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 22:54:27.392153 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-1-root
uuid: "de2c9d4c5894484e95281c4b60fa46ef"
format_stamp: "Formatted at 2026-08-09 22:54:27 on dist-test-slave-gp6n"
I20260809 22:54:27.392441 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:27.415072 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:27.416291 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:27.417837 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:54:27.420038 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:27.420246 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:27.420483 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:27.420640 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:27.463730 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.130:42601
I20260809 22:54:27.463866 8358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.130:42601 every 8 connection(s)
I20260809 22:54:27.468369 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:27.476208 8363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:27.476594 8364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:27.479962 5542 server_base.cc:1034] running on GCE node
W20260809 22:54:27.480194 8367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:27.481283 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:27.481549 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:27.481734 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316067481722 us; error 0 us; skew 500 ppm
I20260809 22:54:27.482312 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:27.482623 8360 heartbeater.cc:347] Connected to a master server at 127.5.105.190:39521
I20260809 22:54:27.483034 8360 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:27.483878 8360 heartbeater.cc:511] Master 127.5.105.190:39521 requested a full tablet report, sending...
I20260809 22:54:27.484905 5542 webserver.cc:460] Webserver started at http://127.5.105.131:46689/ using document root <none> and password file <none>
I20260809 22:54:27.485538 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:27.485795 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:27.486106 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:27.486142 8129 ts_manager.cc:194] Registered new tserver with Master: de2c9d4c5894484e95281c4b60fa46ef (127.5.105.130:42601)
I20260809 22:54:27.487880 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-2-root/instance:
uuid: "8c1fa709fbb04dfda4addecd77dad281"
format_stamp: "Formatted at 2026-08-09 22:54:27 on dist-test-slave-gp6n"
I20260809 22:54:27.488466 8129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55478
I20260809 22:54:27.493571 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 22:54:27.497090 8372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:27.497872 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 22:54:27.498167 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-2-root
uuid: "8c1fa709fbb04dfda4addecd77dad281"
format_stamp: "Formatted at 2026-08-09 22:54:27 on dist-test-slave-gp6n"
I20260809 22:54:27.498489 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:27.515295 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:27.516520 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:27.517951 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:54:27.520210 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:27.520433 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:27.520663 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:27.520854 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:27.561709 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.131:45383
I20260809 22:54:27.561812 8443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.131:45383 every 8 connection(s)
I20260809 22:54:27.577677 8444 heartbeater.cc:347] Connected to a master server at 127.5.105.190:39521
I20260809 22:54:27.578125 8444 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:27.578924 8444 heartbeater.cc:511] Master 127.5.105.190:39521 requested a full tablet report, sending...
I20260809 22:54:27.580821 8129 ts_manager.cc:194] Registered new tserver with Master: 8c1fa709fbb04dfda4addecd77dad281 (127.5.105.131:45383)
I20260809 22:54:27.581707 5542 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016840192s
I20260809 22:54:27.582851 8129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55494
I20260809 22:54:27.619850 8129 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260809 22:54:27.621513 8129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:54:27.679498 8319 tablet_service.cc:1467] Processing CreateTablet for tablet de84a26d01e343d8afacbc5d62ea323b (DEFAULT_TABLE table=test-table [id=08d56150a20541b9826631632eeec391]), partition=
I20260809 22:54:27.680817 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de84a26d01e343d8afacbc5d62ea323b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:27.680958 8232 tablet_service.cc:1467] Processing CreateTablet for tablet de84a26d01e343d8afacbc5d62ea323b (DEFAULT_TABLE table=test-table [id=08d56150a20541b9826631632eeec391]), partition=
I20260809 22:54:27.682054 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de84a26d01e343d8afacbc5d62ea323b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:27.684639 8404 tablet_service.cc:1467] Processing CreateTablet for tablet de84a26d01e343d8afacbc5d62ea323b (DEFAULT_TABLE table=test-table [id=08d56150a20541b9826631632eeec391]), partition=
I20260809 22:54:27.685644 8404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de84a26d01e343d8afacbc5d62ea323b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:27.699939 8461 tablet_bootstrap.cc:492] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587: Bootstrap starting.
I20260809 22:54:27.704404 8462 tablet_bootstrap.cc:492] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef: Bootstrap starting.
I20260809 22:54:27.708859 8462 tablet_bootstrap.cc:654] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:27.708971 8461 tablet_bootstrap.cc:654] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:27.711172 8463 tablet_bootstrap.cc:492] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Bootstrap starting.
I20260809 22:54:27.714905 8461 tablet_bootstrap.cc:492] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587: No bootstrap required, opened a new log
I20260809 22:54:27.715212 8462 tablet_bootstrap.cc:492] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef: No bootstrap required, opened a new log
I20260809 22:54:27.715401 8461 ts_tablet_manager.cc:1397] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260809 22:54:27.715790 8462 ts_tablet_manager.cc:1397] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260809 22:54:27.718082 8463 tablet_bootstrap.cc:654] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:27.718309 8461 raft_consensus.cc:348] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c9d4c5894484e95281c4b60fa46ef" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42601 } } peers { permanent_uuid: "8c1fa709fbb04dfda4addecd77dad281" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45383 } } peers { permanent_uuid: "ada126cc77e443318e820939d4e82587" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 44943 } }
I20260809 22:54:27.719138 8461 raft_consensus.cc:374] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:27.718715 8462 raft_consensus.cc:348] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c9d4c5894484e95281c4b60fa46ef" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42601 } } peers { permanent_uuid: "8c1fa709fbb04dfda4addecd77dad281" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45383 } } peers { permanent_uuid: "ada126cc77e443318e820939d4e82587" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 44943 } }
I20260809 22:54:27.719491 8461 raft_consensus.cc:729] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada126cc77e443318e820939d4e82587, State: Initialized, Role: FOLLOWER
I20260809 22:54:27.719646 8462 raft_consensus.cc:374] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:27.719977 8462 raft_consensus.cc:729] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2c9d4c5894484e95281c4b60fa46ef, State: Initialized, Role: FOLLOWER
I20260809 22:54:27.720645 8462 consensus_queue.cc:260] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c9d4c5894484e95281c4b60fa46ef" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42601 } } peers { permanent_uuid: "8c1fa709fbb04dfda4addecd77dad281" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45383 } } peers { permanent_uuid: "ada126cc77e443318e820939d4e82587" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 44943 } }
I20260809 22:54:27.720312 8461 consensus_queue.cc:260] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c9d4c5894484e95281c4b60fa46ef" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42601 } } peers { permanent_uuid: "8c1fa709fbb04dfda4addecd77dad281" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45383 } } peers { permanent_uuid: "ada126cc77e443318e820939d4e82587" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 44943 } }
I20260809 22:54:27.723258 8462 ts_tablet_manager.cc:1428] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260809 22:54:27.723105 8360 heartbeater.cc:503] Master 127.5.105.190:39521 was elected leader, sending a full tablet report...
W20260809 22:54:27.723604 8462 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20260809 22:54:27.725701 8272 heartbeater.cc:503] Master 127.5.105.190:39521 was elected leader, sending a full tablet report...
I20260809 22:54:27.726147 8461 ts_tablet_manager.cc:1428] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
W20260809 22:54:27.726459 8461 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20260809 22:54:27.727718 8463 tablet_bootstrap.cc:492] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: No bootstrap required, opened a new log
I20260809 22:54:27.728204 8463 ts_tablet_manager.cc:1397] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260809 22:54:27.731156 8463 raft_consensus.cc:348] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c9d4c5894484e95281c4b60fa46ef" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42601 } } peers { permanent_uuid: "8c1fa709fbb04dfda4addecd77dad281" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45383 } } peers { permanent_uuid: "ada126cc77e443318e820939d4e82587" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 44943 } }
I20260809 22:54:27.731884 8463 raft_consensus.cc:374] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:27.732102 8463 raft_consensus.cc:729] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1fa709fbb04dfda4addecd77dad281, State: Initialized, Role: FOLLOWER
I20260809 22:54:27.732563 8463 consensus_queue.cc:260] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c9d4c5894484e95281c4b60fa46ef" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42601 } } peers { permanent_uuid: "8c1fa709fbb04dfda4addecd77dad281" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45383 } } peers { permanent_uuid: "ada126cc77e443318e820939d4e82587" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 44943 } }
I20260809 22:54:27.734375 8444 heartbeater.cc:503] Master 127.5.105.190:39521 was elected leader, sending a full tablet report...
I20260809 22:54:27.734639 8463 ts_tablet_manager.cc:1428] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260809 22:54:27.734921 8463 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
W20260809 22:54:27.815492 8445 tablet.cc:2369] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:54:27.880146 8467 raft_consensus.cc:482] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:54:27.880667 8467 raft_consensus.cc:504] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c9d4c5894484e95281c4b60fa46ef" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42601 } } peers { permanent_uuid: "8c1fa709fbb04dfda4addecd77dad281" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45383 } } peers { permanent_uuid: "ada126cc77e443318e820939d4e82587" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 44943 } }
I20260809 22:54:27.882972 8467 leader_election.cc:288] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c1fa709fbb04dfda4addecd77dad281 (127.5.105.131:45383), ada126cc77e443318e820939d4e82587 (127.5.105.129:44943)
I20260809 22:54:27.896157 8415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de84a26d01e343d8afacbc5d62ea323b" candidate_uuid: "de2c9d4c5894484e95281c4b60fa46ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1fa709fbb04dfda4addecd77dad281" is_pre_election: true
I20260809 22:54:27.897146 8415 raft_consensus.cc:2454] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2c9d4c5894484e95281c4b60fa46ef in term 0.
I20260809 22:54:27.897820 8243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de84a26d01e343d8afacbc5d62ea323b" candidate_uuid: "de2c9d4c5894484e95281c4b60fa46ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada126cc77e443318e820939d4e82587" is_pre_election: true
I20260809 22:54:27.898584 8243 raft_consensus.cc:2454] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2c9d4c5894484e95281c4b60fa46ef in term 0.
I20260809 22:54:27.898486 8287 leader_election.cc:302] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c1fa709fbb04dfda4addecd77dad281, de2c9d4c5894484e95281c4b60fa46ef; no voters:
I20260809 22:54:27.899511 8467 raft_consensus.cc:2781] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:54:27.899880 8467 raft_consensus.cc:482] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:54:27.900158 8467 raft_consensus.cc:3037] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:27.905190 8467 raft_consensus.cc:504] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c9d4c5894484e95281c4b60fa46ef" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42601 } } peers { permanent_uuid: "8c1fa709fbb04dfda4addecd77dad281" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45383 } } peers { permanent_uuid: "ada126cc77e443318e820939d4e82587" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 44943 } }
I20260809 22:54:27.906807 8467 leader_election.cc:288] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [CANDIDATE]: Term 1 election: Requested vote from peers 8c1fa709fbb04dfda4addecd77dad281 (127.5.105.131:45383), ada126cc77e443318e820939d4e82587 (127.5.105.129:44943)
I20260809 22:54:27.907622 8415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de84a26d01e343d8afacbc5d62ea323b" candidate_uuid: "de2c9d4c5894484e95281c4b60fa46ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1fa709fbb04dfda4addecd77dad281"
I20260809 22:54:27.907781 8243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de84a26d01e343d8afacbc5d62ea323b" candidate_uuid: "de2c9d4c5894484e95281c4b60fa46ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada126cc77e443318e820939d4e82587"
I20260809 22:54:27.908125 8415 raft_consensus.cc:3037] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:27.908361 8243 raft_consensus.cc:3037] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:27.913367 8415 raft_consensus.cc:2454] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2c9d4c5894484e95281c4b60fa46ef in term 1.
I20260809 22:54:27.913471 8243 raft_consensus.cc:2454] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2c9d4c5894484e95281c4b60fa46ef in term 1.
I20260809 22:54:27.914337 8287 leader_election.cc:302] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c1fa709fbb04dfda4addecd77dad281, de2c9d4c5894484e95281c4b60fa46ef; no voters:
I20260809 22:54:27.915084 8467 raft_consensus.cc:2781] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:27.915971 8467 raft_consensus.cc:686] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [term 1 LEADER]: Becoming Leader. State: Replica: de2c9d4c5894484e95281c4b60fa46ef, State: Running, Role: LEADER
I20260809 22:54:27.916667 8467 consensus_queue.cc:237] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c9d4c5894484e95281c4b60fa46ef" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42601 } } peers { permanent_uuid: "8c1fa709fbb04dfda4addecd77dad281" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45383 } } peers { permanent_uuid: "ada126cc77e443318e820939d4e82587" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 44943 } }
I20260809 22:54:27.925057 8127 catalog_manager.cc:5495] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef reported cstate change: term changed from 0 to 1, leader changed from <none> to de2c9d4c5894484e95281c4b60fa46ef (127.5.105.130). New cstate: current_term: 1 leader_uuid: "de2c9d4c5894484e95281c4b60fa46ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c9d4c5894484e95281c4b60fa46ef" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c1fa709fbb04dfda4addecd77dad281" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada126cc77e443318e820939d4e82587" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 44943 } health_report { overall_health: UNKNOWN } } }
I20260809 22:54:28.013739 8404 tablet_service.cc:1514] Processing DeleteTablet for tablet de84a26d01e343d8afacbc5d62ea323b with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51026
I20260809 22:54:28.015430 8474 tablet_replica.cc:330] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: stopping tablet replica
I20260809 22:54:28.016019 8474 raft_consensus.cc:2229] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:28.016395 8474 raft_consensus.cc:2258] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:28.018630 8474 ts_tablet_manager.cc:1905] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 22:54:28.029021 8474 ts_tablet_manager.cc:1918] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 0.0
I20260809 22:54:28.029379 8474 log.cc:1196] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Deleting WAL directory at /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-2-root/wals/de84a26d01e343d8afacbc5d62ea323b
I20260809 22:54:28.033586 8475 ts_tablet_manager.cc:927] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Initiating tablet copy from peer de2c9d4c5894484e95281c4b60fa46ef (127.5.105.130:42601)
I20260809 22:54:28.035516 8475 tablet_copy_client.cc:316] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: tablet copy: Beginning tablet copy session from remote peer at address 127.5.105.130:42601
I20260809 22:54:28.044961 8340 tablet_copy_service.cc:140] P de2c9d4c5894484e95281c4b60fa46ef: Received BeginTabletCopySession request for tablet de84a26d01e343d8afacbc5d62ea323b from peer 8c1fa709fbb04dfda4addecd77dad281 ({username='slave'} at 127.0.0.1:60928)
I20260809 22:54:28.045507 8340 tablet_copy_service.cc:161] P de2c9d4c5894484e95281c4b60fa46ef: Beginning new tablet copy session on tablet de84a26d01e343d8afacbc5d62ea323b from peer 8c1fa709fbb04dfda4addecd77dad281 at {username='slave'} at 127.0.0.1:60928: session id = 8c1fa709fbb04dfda4addecd77dad281-de84a26d01e343d8afacbc5d62ea323b
I20260809 22:54:28.051240 8340 tablet_copy_source_session.cc:205] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 22:54:28.053650 8475 ts_tablet_manager.cc:1905] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260809 22:54:28.063275 8475 ts_tablet_manager.cc:1918] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 0.0
I20260809 22:54:28.063848 8475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de84a26d01e343d8afacbc5d62ea323b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:28.068605 8475 tablet_copy_client.cc:791] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: tablet copy: Starting download of 0 data blocks...
I20260809 22:54:28.069092 8475 tablet_copy_client.cc:655] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: tablet copy: Starting download of 1 WAL segments...
I20260809 22:54:28.070751 8127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55520:
table { table_id: "08d56150a20541b9826631632eeec391" table_name: "test-table" }
I20260809 22:54:28.071465 8127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55520:
table { table_id: "08d56150a20541b9826631632eeec391" table_name: "test-table" }
I20260809 22:54:28.073096 8475 tablet_copy_client.cc:531] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 22:54:28.079770 8475 tablet_bootstrap.cc:492] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Bootstrap starting.
I20260809 22:54:28.081784 8127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774: Sending DeleteTablet for 3 replicas of tablet de84a26d01e343d8afacbc5d62ea323b
I20260809 22:54:28.083845 8319 tablet_service.cc:1514] Processing DeleteTablet for tablet de84a26d01e343d8afacbc5d62ea323b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:54:28 UTC) from {username='slave'} at 127.0.0.1:60904
I20260809 22:54:28.084420 8404 tablet_service.cc:1514] Processing DeleteTablet for tablet de84a26d01e343d8afacbc5d62ea323b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:54:28 UTC) from {username='slave'} at 127.0.0.1:51000
I20260809 22:54:28.085202 8232 tablet_service.cc:1514] Processing DeleteTablet for tablet de84a26d01e343d8afacbc5d62ea323b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:54:28 UTC) from {username='slave'} at 127.0.0.1:41362
W20260809 22:54:28.086642 8111 catalog_manager.cc:4821] TS 8c1fa709fbb04dfda4addecd77dad281 (127.5.105.131:45383): delete failed for tablet de84a26d01e343d8afacbc5d62ea323b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de84a26d01e343d8afacbc5d62ea323b already in progress: copying tablet
I20260809 22:54:28.087774 8478 tablet_replica.cc:330] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587: stopping tablet replica
I20260809 22:54:28.088794 8478 raft_consensus.cc:2229] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:28.089548 8478 raft_consensus.cc:2258] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:28.090809 8477 tablet_replica.cc:330] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef: stopping tablet replica
I20260809 22:54:28.091775 8477 raft_consensus.cc:2229] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:28.092449 8477 pending_rounds.cc:62] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef: Trying to abort 1 pending ops.
I20260809 22:54:28.092761 8478 ts_tablet_manager.cc:1905] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:54:28.092808 8477 pending_rounds.cc:68] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316750614193147904 op_type: NO_OP noop_request { }
I20260809 22:54:28.093552 8477 raft_consensus.cc:2866] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260809 22:54:28.093961 8477 raft_consensus.cc:2258] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:28.096148 8475 tablet_bootstrap.cc:492] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260809 22:54:28.097076 8475 tablet_bootstrap.cc:492] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Bootstrap complete.
I20260809 22:54:28.097157 8477 ts_tablet_manager.cc:1905] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:54:28.097786 8475 ts_tablet_manager.cc:1397] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260809 22:54:28.100708 8475 raft_consensus.cc:348] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c9d4c5894484e95281c4b60fa46ef" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42601 } } peers { permanent_uuid: "8c1fa709fbb04dfda4addecd77dad281" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45383 } } peers { permanent_uuid: "ada126cc77e443318e820939d4e82587" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 44943 } }
I20260809 22:54:28.101732 8475 raft_consensus.cc:729] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1fa709fbb04dfda4addecd77dad281, State: Initialized, Role: FOLLOWER
I20260809 22:54:28.102514 8475 consensus_queue.cc:260] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c9d4c5894484e95281c4b60fa46ef" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42601 } } peers { permanent_uuid: "8c1fa709fbb04dfda4addecd77dad281" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 45383 } } peers { permanent_uuid: "ada126cc77e443318e820939d4e82587" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 44943 } }
I20260809 22:54:28.104976 8475 ts_tablet_manager.cc:1428] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
W20260809 22:54:28.105223 8475 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20260809 22:54:28.105748 8478 ts_tablet_manager.cc:1918] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260809 22:54:28.106139 8478 log.cc:1196] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587: Deleting WAL directory at /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-0-root/wals/de84a26d01e343d8afacbc5d62ea323b
I20260809 22:54:28.106806 8478 ts_tablet_manager.cc:1939] T de84a26d01e343d8afacbc5d62ea323b P ada126cc77e443318e820939d4e82587: Deleting consensus metadata
I20260809 22:54:28.109377 8113 catalog_manager.cc:4841] TS ada126cc77e443318e820939d4e82587 (127.5.105.129:44943): tablet de84a26d01e343d8afacbc5d62ea323b (table test-table [id=08d56150a20541b9826631632eeec391]) successfully deleted
I20260809 22:54:28.110086 8404 tablet_service.cc:1514] Processing DeleteTablet for tablet de84a26d01e343d8afacbc5d62ea323b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:54:28 UTC) from {username='slave'} at 127.0.0.1:51000
I20260809 22:54:28.111191 8477 ts_tablet_manager.cc:1918] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260809 22:54:28.111599 8477 log.cc:1196] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef: Deleting WAL directory at /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-1-root/wals/de84a26d01e343d8afacbc5d62ea323b
W20260809 22:54:28.111519 8111 catalog_manager.cc:4821] TS 8c1fa709fbb04dfda4addecd77dad281 (127.5.105.131:45383): delete failed for tablet de84a26d01e343d8afacbc5d62ea323b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de84a26d01e343d8afacbc5d62ea323b already in progress: copying tablet
I20260809 22:54:28.112525 8477 ts_tablet_manager.cc:1939] T de84a26d01e343d8afacbc5d62ea323b P de2c9d4c5894484e95281c4b60fa46ef: Deleting consensus metadata
I20260809 22:54:28.113631 8114 catalog_manager.cc:4841] TS de2c9d4c5894484e95281c4b60fa46ef (127.5.105.130:42601): tablet de84a26d01e343d8afacbc5d62ea323b (table test-table [id=08d56150a20541b9826631632eeec391]) successfully deleted
I20260809 22:54:28.157636 8340 tablet_copy_service.cc:342] P de2c9d4c5894484e95281c4b60fa46ef: Request end of tablet copy session 8c1fa709fbb04dfda4addecd77dad281-de84a26d01e343d8afacbc5d62ea323b received from {username='slave'} at 127.0.0.1:60928
I20260809 22:54:28.158095 8340 tablet_copy_service.cc:433] P de2c9d4c5894484e95281c4b60fa46ef: ending tablet copy session 8c1fa709fbb04dfda4addecd77dad281-de84a26d01e343d8afacbc5d62ea323b on tablet de84a26d01e343d8afacbc5d62ea323b with peer 8c1fa709fbb04dfda4addecd77dad281
I20260809 22:54:28.161685 8404 tablet_service.cc:1514] Processing DeleteTablet for tablet de84a26d01e343d8afacbc5d62ea323b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:54:28 UTC) from {username='slave'} at 127.0.0.1:51000
I20260809 22:54:28.162395 8474 tablet_replica.cc:330] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: stopping tablet replica
I20260809 22:54:28.163139 8474 raft_consensus.cc:2229] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:28.163574 8474 pending_rounds.cc:62] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Trying to abort 1 pending ops.
I20260809 22:54:28.163827 8474 pending_rounds.cc:68] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316750614193147904 op_type: NO_OP noop_request { }
I20260809 22:54:28.164212 8474 raft_consensus.cc:2866] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 22:54:28.164546 8474 raft_consensus.cc:2258] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:28.166481 8474 ts_tablet_manager.cc:1905] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:54:28.176221 8474 ts_tablet_manager.cc:1918] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260809 22:54:28.176496 8474 log.cc:1196] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Deleting WAL directory at /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786316013627029-5542-0/minicluster-data/ts-2-root/wals/de84a26d01e343d8afacbc5d62ea323b
I20260809 22:54:28.177016 8474 ts_tablet_manager.cc:1939] T de84a26d01e343d8afacbc5d62ea323b P 8c1fa709fbb04dfda4addecd77dad281: Deleting consensus metadata
I20260809 22:54:28.179373 8111 catalog_manager.cc:4841] TS 8c1fa709fbb04dfda4addecd77dad281 (127.5.105.131:45383): tablet de84a26d01e343d8afacbc5d62ea323b (table test-table [id=08d56150a20541b9826631632eeec391]) successfully deleted
I20260809 22:54:28.222498 5542 tablet_server.cc:178] TabletServer@127.5.105.129:0 shutting down...
I20260809 22:54:28.244145 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:28.260766 5542 tablet_server.cc:194] TabletServer@127.5.105.129:0 shutdown complete.
I20260809 22:54:28.268729 5542 tablet_server.cc:178] TabletServer@127.5.105.130:0 shutting down...
I20260809 22:54:28.285969 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:28.303082 5542 tablet_server.cc:194] TabletServer@127.5.105.130:0 shutdown complete.
I20260809 22:54:28.311729 5542 tablet_server.cc:178] TabletServer@127.5.105.131:0 shutting down...
I20260809 22:54:28.329581 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:28.346524 5542 tablet_server.cc:194] TabletServer@127.5.105.131:0 shutdown complete.
I20260809 22:54:28.355486 5542 master.cc:537] Master@127.5.105.190:39521 shutting down...
I20260809 22:54:28.372493 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:28.372978 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:28.373229 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4a4ee12ee8e74c5199546233ebe6d774: stopping tablet replica
I20260809 22:54:28.382164 5542 master.cc:552] Master@127.5.105.190:39521 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (1346 ms)
[----------] 4 tests from TsTabletManagerITest (44237 ms total)
[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20260809 22:54:28.407130 5542 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.5.105.190:32907
I20260809 22:54:28.408062 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:28.412609 8485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:28.413928 8486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:28.415652 8488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:28.416227 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:28.417135 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:28.417295 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:28.417409 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316068417401 us; error 0 us; skew 500 ppm
I20260809 22:54:28.417840 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:28.420132 5542 webserver.cc:460] Webserver started at http://127.5.105.190:34075/ using document root <none> and password file <none>
I20260809 22:54:28.420564 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:28.420706 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:28.420907 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:28.421900 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/master-0-root/instance:
uuid: "9e09628a8c874bbaadba77c7821d332b"
format_stamp: "Formatted at 2026-08-09 22:54:28 on dist-test-slave-gp6n"
I20260809 22:54:28.426172 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260809 22:54:28.429461 8493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:28.430267 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 22:54:28.430670 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/master-0-root
uuid: "9e09628a8c874bbaadba77c7821d332b"
format_stamp: "Formatted at 2026-08-09 22:54:28 on dist-test-slave-gp6n"
I20260809 22:54:28.431005 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:28.451820 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:28.452893 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:28.492034 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.190:32907
I20260809 22:54:28.492132 8550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.190:32907 every 8 connection(s)
I20260809 22:54:28.495913 8551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:28.505767 8551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b: Bootstrap starting.
I20260809 22:54:28.509825 8551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:28.513615 8551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b: No bootstrap required, opened a new log
I20260809 22:54:28.515466 8551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e09628a8c874bbaadba77c7821d332b" member_type: VOTER }
I20260809 22:54:28.515846 8551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:28.516090 8551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e09628a8c874bbaadba77c7821d332b, State: Initialized, Role: FOLLOWER
I20260809 22:54:28.516623 8551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e09628a8c874bbaadba77c7821d332b" member_type: VOTER }
I20260809 22:54:28.517056 8551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:54:28.517278 8551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:54:28.517508 8551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:28.521471 8551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e09628a8c874bbaadba77c7821d332b" member_type: VOTER }
I20260809 22:54:28.521965 8551 leader_election.cc:302] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e09628a8c874bbaadba77c7821d332b; no voters:
I20260809 22:54:28.523100 8551 leader_election.cc:288] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:54:28.523458 8555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:28.524808 8555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [term 1 LEADER]: Becoming Leader. State: Replica: 9e09628a8c874bbaadba77c7821d332b, State: Running, Role: LEADER
I20260809 22:54:28.525449 8555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e09628a8c874bbaadba77c7821d332b" member_type: VOTER }
I20260809 22:54:28.526007 8551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:54:28.528406 8556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e09628a8c874bbaadba77c7821d332b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e09628a8c874bbaadba77c7821d332b" member_type: VOTER } }
I20260809 22:54:28.529213 8556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:28.528412 8557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e09628a8c874bbaadba77c7821d332b. Latest consensus state: current_term: 1 leader_uuid: "9e09628a8c874bbaadba77c7821d332b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e09628a8c874bbaadba77c7821d332b" member_type: VOTER } }
I20260809 22:54:28.529776 8557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:28.533603 8562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:54:28.541374 8562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:54:28.542455 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 22:54:28.549633 8562 catalog_manager.cc:1347] Generated new cluster ID: 892eeecfb6ae4f629a38ac56bcc1a546
I20260809 22:54:28.549923 8562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:54:28.586061 8562 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:54:28.587780 8562 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:54:28.597862 8562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b: Generated new TSK 0
I20260809 22:54:28.598479 8562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:54:28.608781 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:28.614518 8577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:28.615813 8578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:28.617230 8580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:28.617496 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:28.618340 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:28.618582 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:28.618698 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316068618689 us; error 0 us; skew 500 ppm
I20260809 22:54:28.619154 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:28.621438 5542 webserver.cc:460] Webserver started at http://127.5.105.129:35175/ using document root <none> and password file <none>
I20260809 22:54:28.621865 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:28.622023 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:28.622219 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:28.623337 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root/instance:
uuid: "7bcf8303e7294a62b4d8e87d611a9fc9"
format_stamp: "Formatted at 2026-08-09 22:54:28 on dist-test-slave-gp6n"
I20260809 22:54:28.627853 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 22:54:28.630971 8586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:28.631754 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260809 22:54:28.632071 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "7bcf8303e7294a62b4d8e87d611a9fc9"
format_stamp: "Formatted at 2026-08-09 22:54:28 on dist-test-slave-gp6n"
I20260809 22:54:28.632336 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:28.661319 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:28.662349 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:28.667299 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:28.667666 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260809 22:54:28.668071 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:28.668351 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:28.729018 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:41125
I20260809 22:54:28.729123 8662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:41125 every 8 connection(s)
I20260809 22:54:28.733736 8663 heartbeater.cc:347] Connected to a master server at 127.5.105.190:32907
I20260809 22:54:28.734136 8663 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:28.734964 8663 heartbeater.cc:511] Master 127.5.105.190:32907 requested a full tablet report, sending...
I20260809 22:54:28.736930 8511 ts_manager.cc:194] Registered new tserver with Master: 7bcf8303e7294a62b4d8e87d611a9fc9 (127.5.105.129:41125)
I20260809 22:54:28.737116 5542 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004884371s
I20260809 22:54:28.739768 8511 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46766
I20260809 22:54:28.774067 8628 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260809 22:54:28.775465 8628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:28.787050 8674 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: Bootstrap starting.
I20260809 22:54:28.791260 8674 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:28.795205 8674 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: No bootstrap required, opened a new log
I20260809 22:54:28.795603 8674 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260809 22:54:28.797406 8674 raft_consensus.cc:348] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 41125 } }
I20260809 22:54:28.797832 8674 raft_consensus.cc:374] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:28.798063 8674 raft_consensus.cc:729] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bcf8303e7294a62b4d8e87d611a9fc9, State: Initialized, Role: FOLLOWER
I20260809 22:54:28.798659 8674 consensus_queue.cc:260] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 41125 } }
I20260809 22:54:28.799139 8674 raft_consensus.cc:388] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:54:28.799372 8674 raft_consensus.cc:482] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:54:28.799626 8674 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:28.804388 8674 raft_consensus.cc:504] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 41125 } }
I20260809 22:54:28.804946 8674 leader_election.cc:302] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bcf8303e7294a62b4d8e87d611a9fc9; no voters:
I20260809 22:54:28.806154 8674 leader_election.cc:288] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:54:28.806530 8676 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:28.808216 8663 heartbeater.cc:503] Master 127.5.105.190:32907 was elected leader, sending a full tablet report...
I20260809 22:54:28.808462 8674 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260809 22:54:28.808990 8676 raft_consensus.cc:686] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 1 LEADER]: Becoming Leader. State: Replica: 7bcf8303e7294a62b4d8e87d611a9fc9, State: Running, Role: LEADER
I20260809 22:54:28.809602 8676 consensus_queue.cc:237] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 41125 } }
I20260809 22:54:28.814680 8678 tablet_replica.cc:439] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: TxnStatusTablet state changed. Reason: New leader 7bcf8303e7294a62b4d8e87d611a9fc9. Latest consensus state: current_term: 1 leader_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" committed_config { opid_index: -1 peers { permanent_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 41125 } } }
I20260809 22:54:28.814566 8677 tablet_replica.cc:439] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" committed_config { opid_index: -1 peers { permanent_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 41125 } } }
I20260809 22:54:28.815395 8678 tablet_replica.cc:442] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:54:28.815493 8677 tablet_replica.cc:442] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:54:28.817044 8683 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 22:54:28.817552 8683 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260809 22:54:28.818037 8511 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260809 22:54:28.822161 8511 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260809 22:54:28.853092 5542 tablet_server.cc:178] TabletServer@127.5.105.129:0 shutting down...
I20260809 22:54:28.874826 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:28.875952 5542 tablet_replica.cc:330] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: stopping tablet replica
I20260809 22:54:28.876431 5542 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:28.876885 5542 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:28.898516 5542 tablet_server.cc:194] TabletServer@127.5.105.129:0 shutdown complete.
I20260809 22:54:28.910518 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:28.916676 8686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:28.918120 8687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:28.920125 8689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:28.920343 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:28.921466 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:28.921679 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:28.921876 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316068921848 us; error 0 us; skew 500 ppm
I20260809 22:54:28.922513 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:28.925416 5542 webserver.cc:460] Webserver started at http://127.5.105.129:35175/ using document root <none> and password file <none>
I20260809 22:54:28.925978 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:28.926163 5542 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260809 22:54:28.930547 5542 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:54:28.933707 8694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:28.934628 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:54:28.934911 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "7bcf8303e7294a62b4d8e87d611a9fc9"
format_stamp: "Formatted at 2026-08-09 22:54:28 on dist-test-slave-gp6n"
I20260809 22:54:28.935210 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:28.955323 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:28.956342 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:28.964216 8703 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260809 22:54:28.973351 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 22:54:28.973758 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.002s sys 0.000s
I20260809 22:54:28.974045 5542 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 22:54:28.978927 5542 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 22:54:28.979215 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20260809 22:54:28.979245 8703 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: Bootstrap starting.
I20260809 22:54:29.019899 8703 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 22:54:29.021006 8703 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: Bootstrap complete.
I20260809 22:54:29.021754 8703 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: Time spent bootstrapping tablet: real 0.043s user 0.041s sys 0.000s
I20260809 22:54:29.024322 8703 raft_consensus.cc:348] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 41125 } }
I20260809 22:54:29.025002 8703 raft_consensus.cc:729] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bcf8303e7294a62b4d8e87d611a9fc9, State: Initialized, Role: FOLLOWER
I20260809 22:54:29.025759 8703 consensus_queue.cc:260] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 41125 } }
I20260809 22:54:29.026609 8703 raft_consensus.cc:388] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:54:29.027026 8703 raft_consensus.cc:482] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:54:29.027575 8703 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:54:29.038139 8703 raft_consensus.cc:504] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 41125 } }
I20260809 22:54:29.038990 8703 leader_election.cc:302] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bcf8303e7294a62b4d8e87d611a9fc9; no voters:
I20260809 22:54:29.046975 8703 leader_election.cc:288] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 22:54:29.047406 8722 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 22:54:29.052958 8722 raft_consensus.cc:686] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 2 LEADER]: Becoming Leader. State: Replica: 7bcf8303e7294a62b4d8e87d611a9fc9, State: Running, Role: LEADER
I20260809 22:54:29.054096 8722 consensus_queue.cc:237] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 41125 } }
I20260809 22:54:29.056345 8703 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9: Time spent starting tablet: real 0.034s user 0.019s sys 0.008s
I20260809 22:54:29.060005 8732 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 7bcf8303e7294a62b4d8e87d611a9fc9. Latest consensus state: current_term: 2 leader_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" committed_config { opid_index: -1 peers { permanent_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 41125 } } }
I20260809 22:54:29.060770 8732 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260809 22:54:29.063967 8733 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" committed_config { opid_index: -1 peers { permanent_uuid: "7bcf8303e7294a62b4d8e87d611a9fc9" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 41125 } } }
I20260809 22:54:29.064705 8733 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260809 22:54:29.069839 8738 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 22:54:29.070307 8738 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 22:54:29.107299 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:41125
I20260809 22:54:29.107421 8782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:41125 every 8 connection(s)
I20260809 22:54:29.113092 8783 heartbeater.cc:347] Connected to a master server at 127.5.105.190:32907
I20260809 22:54:29.113557 8783 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:29.115338 5542 tablet_server.cc:178] TabletServer@127.5.105.129:41125 shutting down...
I20260809 22:54:29.114924 8783 heartbeater.cc:511] Master 127.5.105.190:32907 requested a full tablet report, sending...
I20260809 22:54:29.122819 8511 ts_manager.cc:194] Re-registered known tserver with Master: 7bcf8303e7294a62b4d8e87d611a9fc9 (127.5.105.129:41125)
W20260809 22:54:29.123456 8511 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260809 22:54:29.125661 8511 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46810
I20260809 22:54:29.132611 8783 heartbeater.cc:503] Master 127.5.105.190:32907 was elected leader, sending a full tablet report...
W20260809 22:54:29.136252 8511 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260809 22:54:29.148087 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:29.149087 5542 tablet_replica.cc:330] stopping tablet replica
I20260809 22:54:29.149818 5542 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 2 LEADER]: Raft consensus shutting down.
I20260809 22:54:29.150369 5542 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 7bcf8303e7294a62b4d8e87d611a9fc9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:29.163884 5542 tablet_server.cc:194] TabletServer@127.5.105.129:41125 shutdown complete.
I20260809 22:54:29.175197 5542 master.cc:537] Master@127.5.105.190:32907 shutting down...
I20260809 22:54:29.197645 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:29.198180 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:29.198530 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9e09628a8c874bbaadba77c7821d332b: stopping tablet replica
I20260809 22:54:29.217232 5542 master.cc:552] Master@127.5.105.190:32907 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (835 ms)
[ RUN ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20260809 22:54:29.244220 5542 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.5.105.190:33907
I20260809 22:54:29.245368 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:29.251361 8789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:29.251777 8790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:29.255055 8794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:29.255810 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:29.256669 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:29.256851 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:29.257069 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316069257027 us; error 0 us; skew 500 ppm
I20260809 22:54:29.257670 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:29.261019 5542 webserver.cc:460] Webserver started at http://127.5.105.190:40863/ using document root <none> and password file <none>
I20260809 22:54:29.261595 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:29.261808 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:29.262116 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:29.263407 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/master-0-root/instance:
uuid: "792365c850684f248be8d3b45983de4d"
format_stamp: "Formatted at 2026-08-09 22:54:29 on dist-test-slave-gp6n"
I20260809 22:54:29.268512 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 22:54:29.272593 8799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:29.273516 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 22:54:29.273847 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/master-0-root
uuid: "792365c850684f248be8d3b45983de4d"
format_stamp: "Formatted at 2026-08-09 22:54:29 on dist-test-slave-gp6n"
I20260809 22:54:29.274197 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:29.310341 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:29.311721 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:29.359201 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.190:33907
I20260809 22:54:29.359294 8853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.190:33907 every 8 connection(s)
I20260809 22:54:29.364032 8854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:29.373881 8854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d: Bootstrap starting.
I20260809 22:54:29.378382 8854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:29.382364 8854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d: No bootstrap required, opened a new log
I20260809 22:54:29.384231 8854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792365c850684f248be8d3b45983de4d" member_type: VOTER }
I20260809 22:54:29.384652 8854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:29.384899 8854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792365c850684f248be8d3b45983de4d, State: Initialized, Role: FOLLOWER
I20260809 22:54:29.385490 8854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792365c850684f248be8d3b45983de4d" member_type: VOTER }
I20260809 22:54:29.385968 8854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:54:29.386205 8854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:54:29.386489 8854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:29.390812 8854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792365c850684f248be8d3b45983de4d" member_type: VOTER }
I20260809 22:54:29.391310 8854 leader_election.cc:302] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 792365c850684f248be8d3b45983de4d; no voters:
I20260809 22:54:29.392791 8854 leader_election.cc:288] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:54:29.393110 8857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:29.394871 8857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [term 1 LEADER]: Becoming Leader. State: Replica: 792365c850684f248be8d3b45983de4d, State: Running, Role: LEADER
I20260809 22:54:29.395602 8857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792365c850684f248be8d3b45983de4d" member_type: VOTER }
I20260809 22:54:29.396361 8854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:54:29.398567 8858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "792365c850684f248be8d3b45983de4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792365c850684f248be8d3b45983de4d" member_type: VOTER } }
I20260809 22:54:29.398684 8859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 792365c850684f248be8d3b45983de4d. Latest consensus state: current_term: 1 leader_uuid: "792365c850684f248be8d3b45983de4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792365c850684f248be8d3b45983de4d" member_type: VOTER } }
I20260809 22:54:29.399082 8858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:29.399284 8859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:29.401494 8863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:54:29.410255 8863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:54:29.424014 8863 catalog_manager.cc:1347] Generated new cluster ID: 7320b572f96947caa4cb5e33fedf32ac
I20260809 22:54:29.424299 8863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:54:29.428910 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 22:54:29.454779 8863 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:54:29.456638 8863 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:54:29.471212 8863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d: Generated new TSK 0
I20260809 22:54:29.471951 8863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:54:29.495452 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:29.501852 8877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:29.503268 8878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:29.506325 5542 server_base.cc:1034] running on GCE node
W20260809 22:54:29.506342 8880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:29.507548 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:29.507768 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:29.507963 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316069507947 us; error 0 us; skew 500 ppm
I20260809 22:54:29.508491 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:29.511366 5542 webserver.cc:460] Webserver started at http://127.5.105.129:46439/ using document root <none> and password file <none>
I20260809 22:54:29.511850 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:29.512054 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:29.512319 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:29.513407 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root/instance:
uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97"
format_stamp: "Formatted at 2026-08-09 22:54:29 on dist-test-slave-gp6n"
I20260809 22:54:29.518522 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 22:54:29.522140 8885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:29.523029 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:54:29.523306 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97"
format_stamp: "Formatted at 2026-08-09 22:54:29 on dist-test-slave-gp6n"
I20260809 22:54:29.523561 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:29.566239 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:29.567348 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:29.579869 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:29.580346 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.001s sys 0.000s
I20260809 22:54:29.580787 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:29.581110 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:29.660815 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:43553
I20260809 22:54:29.660977 8960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:43553 every 8 connection(s)
I20260809 22:54:29.665763 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 22:54:29.666044 8962 heartbeater.cc:347] Connected to a master server at 127.5.105.190:33907
I20260809 22:54:29.666478 8962 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:29.667294 8962 heartbeater.cc:511] Master 127.5.105.190:33907 requested a full tablet report, sending...
I20260809 22:54:29.669556 8817 ts_manager.cc:194] Registered new tserver with Master: 4d83a6f2ea6b4d128511a1ffa24b0f97 (127.5.105.129:43553)
W20260809 22:54:29.672328 8965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:29.673277 8817 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54446
W20260809 22:54:29.679445 8966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:29.680539 5542 server_base.cc:1034] running on GCE node
W20260809 22:54:29.680987 8968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:29.681953 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:29.682132 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:29.682341 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316069682311 us; error 0 us; skew 500 ppm
I20260809 22:54:29.682911 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:29.686602 5542 webserver.cc:460] Webserver started at http://127.5.105.130:44937/ using document root <none> and password file <none>
I20260809 22:54:29.687084 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:29.687283 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:29.687562 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:29.688623 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-1-root/instance:
uuid: "0978cd296c604d718a32d2e9bf410f5d"
format_stamp: "Formatted at 2026-08-09 22:54:29 on dist-test-slave-gp6n"
I20260809 22:54:29.693186 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260809 22:54:29.696666 8973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:29.697602 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 22:54:29.697960 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-1-root
uuid: "0978cd296c604d718a32d2e9bf410f5d"
format_stamp: "Formatted at 2026-08-09 22:54:29 on dist-test-slave-gp6n"
I20260809 22:54:29.698305 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:29.727533 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:29.728665 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:29.740265 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:29.740664 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:29.740954 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:29.741210 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:29.812107 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.130:40381
I20260809 22:54:29.812355 9048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.130:40381 every 8 connection(s)
I20260809 22:54:29.817399 9049 heartbeater.cc:347] Connected to a master server at 127.5.105.190:33907
I20260809 22:54:29.817822 9049 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:29.818734 9049 heartbeater.cc:511] Master 127.5.105.190:33907 requested a full tablet report, sending...
I20260809 22:54:29.820900 8817 ts_manager.cc:194] Registered new tserver with Master: 0978cd296c604d718a32d2e9bf410f5d (127.5.105.130:40381)
I20260809 22:54:29.821838 5542 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.006409303s
I20260809 22:54:29.823024 8817 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54466
I20260809 22:54:29.863471 8925 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260809 22:54:29.864989 8925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:29.880470 9061 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: Bootstrap starting.
I20260809 22:54:29.885010 9061 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:29.889005 9061 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: No bootstrap required, opened a new log
I20260809 22:54:29.889398 9061 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 22:54:29.891261 9061 raft_consensus.cc:348] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } }
I20260809 22:54:29.891705 9061 raft_consensus.cc:374] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:29.891937 9061 raft_consensus.cc:729] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d83a6f2ea6b4d128511a1ffa24b0f97, State: Initialized, Role: FOLLOWER
I20260809 22:54:29.892442 9061 consensus_queue.cc:260] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } }
I20260809 22:54:29.892911 9061 raft_consensus.cc:388] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:54:29.893146 9061 raft_consensus.cc:482] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:54:29.893411 9061 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:29.897850 9061 raft_consensus.cc:504] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } }
I20260809 22:54:29.898648 9061 leader_election.cc:302] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d83a6f2ea6b4d128511a1ffa24b0f97; no voters:
I20260809 22:54:29.901032 9061 leader_election.cc:288] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:54:29.901535 9064 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:29.902985 8962 heartbeater.cc:503] Master 127.5.105.190:33907 was elected leader, sending a full tablet report...
I20260809 22:54:29.903847 9061 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260809 22:54:29.905104 9064 raft_consensus.cc:686] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 1 LEADER]: Becoming Leader. State: Replica: 4d83a6f2ea6b4d128511a1ffa24b0f97, State: Running, Role: LEADER
I20260809 22:54:29.905819 9064 consensus_queue.cc:237] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } }
I20260809 22:54:29.909474 9065 tablet_replica.cc:439] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" committed_config { opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } } }
I20260809 22:54:29.910006 9065 tablet_replica.cc:442] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: This TxnStatusTablet replica's current role is: LEADER
W20260809 22:54:29.911966 8817 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260809 22:54:29.911790 9066 tablet_replica.cc:439] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: TxnStatusTablet state changed. Reason: New leader 4d83a6f2ea6b4d128511a1ffa24b0f97. Latest consensus state: current_term: 1 leader_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" committed_config { opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } } }
I20260809 22:54:29.912509 9066 tablet_replica.cc:442] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:54:29.916831 9068 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 22:54:29.917271 9068 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260809 22:54:29.920697 8817 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260809 22:54:29.974291 9073 ts_tablet_manager.cc:927] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d: Initiating tablet copy from peer 4d83a6f2ea6b4d128511a1ffa24b0f97 (127.5.105.129:43553)
I20260809 22:54:29.976183 9073 tablet_copy_client.cc:316] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.105.129:43553
I20260809 22:54:29.986948 8946 tablet_copy_service.cc:140] P 4d83a6f2ea6b4d128511a1ffa24b0f97: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer 0978cd296c604d718a32d2e9bf410f5d ({username='slave'} at 127.0.0.1:52542)
I20260809 22:54:29.987476 8946 tablet_copy_service.cc:161] P 4d83a6f2ea6b4d128511a1ffa24b0f97: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer 0978cd296c604d718a32d2e9bf410f5d at {username='slave'} at 127.0.0.1:52542: session id = 0978cd296c604d718a32d2e9bf410f5d-11111111111111111111111111111111
I20260809 22:54:29.992918 8946 tablet_copy_source_session.cc:205] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 22:54:29.995312 9073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:30.005908 9073 tablet_copy_client.cc:791] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d: tablet copy: Starting download of 0 data blocks...
I20260809 22:54:30.006440 9073 tablet_copy_client.cc:655] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d: tablet copy: Starting download of 1 WAL segments...
I20260809 22:54:30.010026 9073 tablet_copy_client.cc:531] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 22:54:30.016088 9073 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d: Bootstrap starting.
I20260809 22:54:30.039209 9073 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 22:54:30.040259 9073 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d: Bootstrap complete.
I20260809 22:54:30.041038 9073 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.009s
I20260809 22:54:30.044011 9073 raft_consensus.cc:348] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } }
I20260809 22:54:30.044525 9073 raft_consensus.cc:729] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 0978cd296c604d718a32d2e9bf410f5d, State: Initialized, Role: NON_PARTICIPANT
I20260809 22:54:30.045042 9073 consensus_queue.cc:260] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } }
I20260809 22:54:30.046923 9076 tablet_replica.cc:439] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } } }
I20260809 22:54:30.047160 9049 heartbeater.cc:503] Master 127.5.105.190:33907 was elected leader, sending a full tablet report...
I20260809 22:54:30.047410 9073 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d: Time spent starting tablet: real 0.006s user 0.002s sys 0.006s
I20260809 22:54:30.047549 9076 tablet_replica.cc:442] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20260809 22:54:30.049187 8946 tablet_copy_service.cc:342] P 4d83a6f2ea6b4d128511a1ffa24b0f97: Request end of tablet copy session 0978cd296c604d718a32d2e9bf410f5d-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:52542
I20260809 22:54:30.049579 8946 tablet_copy_service.cc:433] P 4d83a6f2ea6b4d128511a1ffa24b0f97: ending tablet copy session 0978cd296c604d718a32d2e9bf410f5d-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer 0978cd296c604d718a32d2e9bf410f5d
W20260809 22:54:30.050091 8817 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260809 22:54:30.054148 8817 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260809 22:54:30.071684 5542 tablet_server.cc:178] TabletServer@127.5.105.129:0 shutting down...
I20260809 22:54:30.095177 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:30.095981 5542 tablet_replica.cc:330] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: stopping tablet replica
I20260809 22:54:30.096606 5542 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:30.097085 5542 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:30.110700 5542 tablet_server.cc:194] TabletServer@127.5.105.129:0 shutdown complete.
I20260809 22:54:30.121404 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:30.128973 9078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:30.129742 9077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:30.131173 9080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:30.132067 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:30.132968 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:30.133179 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:30.133348 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316070133334 us; error 0 us; skew 500 ppm
I20260809 22:54:30.133864 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:30.136749 5542 webserver.cc:460] Webserver started at http://127.5.105.129:46439/ using document root <none> and password file <none>
I20260809 22:54:30.137228 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:30.137434 5542 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260809 22:54:30.141515 5542 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:54:30.144513 9085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:30.145401 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 22:54:30.145768 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97"
format_stamp: "Formatted at 2026-08-09 22:54:29 on dist-test-slave-gp6n"
I20260809 22:54:30.146157 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:30.165017 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:30.166344 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:30.175137 9094 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260809 22:54:30.183085 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 22:54:30.183480 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20260809 22:54:30.183822 5542 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 22:54:30.190615 5542 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 22:54:30.190881 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.007s sys 0.000s
I20260809 22:54:30.191007 9094 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: Bootstrap starting.
I20260809 22:54:30.244302 9094 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 22:54:30.245427 9094 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: Bootstrap complete.
I20260809 22:54:30.246557 9094 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: Time spent bootstrapping tablet: real 0.056s user 0.029s sys 0.025s
I20260809 22:54:30.249037 9094 raft_consensus.cc:348] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } }
I20260809 22:54:30.249502 9094 raft_consensus.cc:729] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d83a6f2ea6b4d128511a1ffa24b0f97, State: Initialized, Role: FOLLOWER
I20260809 22:54:30.250069 9094 consensus_queue.cc:260] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } }
I20260809 22:54:30.250648 9094 raft_consensus.cc:388] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:54:30.251046 9094 raft_consensus.cc:482] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:54:30.251353 9094 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:54:30.256340 9094 raft_consensus.cc:504] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } }
I20260809 22:54:30.256888 9094 leader_election.cc:302] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d83a6f2ea6b4d128511a1ffa24b0f97; no voters:
I20260809 22:54:30.263710 9094 leader_election.cc:288] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 22:54:30.264096 9110 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 22:54:30.276691 9110 raft_consensus.cc:686] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 2 LEADER]: Becoming Leader. State: Replica: 4d83a6f2ea6b4d128511a1ffa24b0f97, State: Running, Role: LEADER
I20260809 22:54:30.277205 9094 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97: Time spent starting tablet: real 0.030s user 0.008s sys 0.021s
I20260809 22:54:30.277658 9110 consensus_queue.cc:237] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } }
I20260809 22:54:30.280854 9112 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" committed_config { opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } } }
I20260809 22:54:30.281376 9112 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260809 22:54:30.287894 9113 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 4d83a6f2ea6b4d128511a1ffa24b0f97. Latest consensus state: current_term: 2 leader_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" committed_config { opid_index: -1 peers { permanent_uuid: "4d83a6f2ea6b4d128511a1ffa24b0f97" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 43553 } } }
I20260809 22:54:30.288617 9113 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260809 22:54:30.290436 9117 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 22:54:30.290895 9117 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 22:54:30.334139 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:43553
I20260809 22:54:30.334223 9172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:43553 every 8 connection(s)
I20260809 22:54:30.341619 5542 tablet_server.cc:178] TabletServer@127.5.105.129:43553 shutting down...
I20260809 22:54:30.341271 9173 heartbeater.cc:347] Connected to a master server at 127.5.105.190:33907
I20260809 22:54:30.342156 9173 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:30.343334 9173 heartbeater.cc:511] Master 127.5.105.190:33907 requested a full tablet report, sending...
I20260809 22:54:30.347950 8817 ts_manager.cc:194] Re-registered known tserver with Master: 4d83a6f2ea6b4d128511a1ffa24b0f97 (127.5.105.129:43553)
W20260809 22:54:30.349620 8817 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260809 22:54:30.351773 8817 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54496
I20260809 22:54:30.355967 9173 heartbeater.cc:503] Master 127.5.105.190:33907 was elected leader, sending a full tablet report...
W20260809 22:54:30.359375 8817 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260809 22:54:30.370000 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:30.370952 5542 tablet_replica.cc:330] stopping tablet replica
I20260809 22:54:30.371531 5542 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 2 LEADER]: Raft consensus shutting down.
I20260809 22:54:30.371975 5542 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 4d83a6f2ea6b4d128511a1ffa24b0f97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:30.383481 5542 tablet_server.cc:194] TabletServer@127.5.105.129:43553 shutdown complete.
I20260809 22:54:30.394851 5542 tablet_server.cc:178] TabletServer@127.5.105.130:0 shutting down...
I20260809 22:54:30.412875 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:30.413746 5542 tablet_replica.cc:330] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d: stopping tablet replica
I20260809 22:54:30.414263 5542 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260809 22:54:30.414726 5542 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 0978cd296c604d718a32d2e9bf410f5d [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260809 22:54:30.434042 5542 tablet_server.cc:194] TabletServer@127.5.105.130:0 shutdown complete.
I20260809 22:54:30.448750 5542 master.cc:537] Master@127.5.105.190:33907 shutting down...
I20260809 22:54:30.469354 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:30.469862 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:30.470182 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P 792365c850684f248be8d3b45983de4d: stopping tablet replica
I20260809 22:54:30.488407 5542 master.cc:552] Master@127.5.105.190:33907 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (1270 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20260809 22:54:30.513829 5542 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.5.105.190:37671
I20260809 22:54:30.514914 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:30.520326 9179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:30.521343 9180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:30.523264 9183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:30.524220 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:30.525010 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:30.525173 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:30.525341 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316070525328 us; error 0 us; skew 500 ppm
I20260809 22:54:30.525889 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:30.528786 5542 webserver.cc:460] Webserver started at http://127.5.105.190:38263/ using document root <none> and password file <none>
I20260809 22:54:30.529259 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:30.529457 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:30.529732 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:30.530886 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786316013627029-5542-0/minicluster-data/master-0-root/instance:
uuid: "fda9adec3cba4c278e24bc01a4625b49"
format_stamp: "Formatted at 2026-08-09 22:54:30 on dist-test-slave-gp6n"
I20260809 22:54:30.535444 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:54:30.538903 9189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:30.539780 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260809 22:54:30.540115 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786316013627029-5542-0/minicluster-data/master-0-root
uuid: "fda9adec3cba4c278e24bc01a4625b49"
format_stamp: "Formatted at 2026-08-09 22:54:30 on dist-test-slave-gp6n"
I20260809 22:54:30.540454 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786316013627029-5542-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786316013627029-5542-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786316013627029-5542-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:30.562482 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:30.563783 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:30.610061 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.190:37671
I20260809 22:54:30.610229 9253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.190:37671 every 8 connection(s)
I20260809 22:54:30.614962 9254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:30.624701 9254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49: Bootstrap starting.
I20260809 22:54:30.628821 9254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:30.632928 9254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49: No bootstrap required, opened a new log
I20260809 22:54:30.634852 9254 raft_consensus.cc:348] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda9adec3cba4c278e24bc01a4625b49" member_type: VOTER }
I20260809 22:54:30.635277 9254 raft_consensus.cc:374] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:30.635520 9254 raft_consensus.cc:729] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda9adec3cba4c278e24bc01a4625b49, State: Initialized, Role: FOLLOWER
I20260809 22:54:30.636034 9254 consensus_queue.cc:260] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda9adec3cba4c278e24bc01a4625b49" member_type: VOTER }
I20260809 22:54:30.636487 9254 raft_consensus.cc:388] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:54:30.636724 9254 raft_consensus.cc:482] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:54:30.636986 9254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:30.642012 9254 raft_consensus.cc:504] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda9adec3cba4c278e24bc01a4625b49" member_type: VOTER }
I20260809 22:54:30.642643 9254 leader_election.cc:302] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fda9adec3cba4c278e24bc01a4625b49; no voters:
I20260809 22:54:30.644026 9254 leader_election.cc:288] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:54:30.644380 9258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:30.646744 9258 raft_consensus.cc:686] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [term 1 LEADER]: Becoming Leader. State: Replica: fda9adec3cba4c278e24bc01a4625b49, State: Running, Role: LEADER
I20260809 22:54:30.647435 9254 sys_catalog.cc:564] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:54:30.647540 9258 consensus_queue.cc:237] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda9adec3cba4c278e24bc01a4625b49" member_type: VOTER }
I20260809 22:54:30.651851 9259 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fda9adec3cba4c278e24bc01a4625b49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda9adec3cba4c278e24bc01a4625b49" member_type: VOTER } }
I20260809 22:54:30.652420 9259 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:30.653606 9260 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fda9adec3cba4c278e24bc01a4625b49. Latest consensus state: current_term: 1 leader_uuid: "fda9adec3cba4c278e24bc01a4625b49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda9adec3cba4c278e24bc01a4625b49" member_type: VOTER } }
I20260809 22:54:30.654122 9260 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:30.665845 9264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:54:30.672447 9264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:54:30.673614 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 22:54:30.682704 9264 catalog_manager.cc:1347] Generated new cluster ID: 3dc19169210c4b43bc561a04d2c79105
I20260809 22:54:30.683001 9264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:54:30.714295 9264 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:54:30.716622 9264 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:54:30.739869 9264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49: Generated new TSK 0
I20260809 22:54:30.740594 9264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:54:30.804301 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:30.810655 9278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:30.812516 9279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:30.815312 9282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:30.816231 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:30.817031 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:30.817230 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:30.817418 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316070817402 us; error 0 us; skew 500 ppm
I20260809 22:54:30.818133 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:30.821142 5542 webserver.cc:460] Webserver started at http://127.5.105.129:42289/ using document root <none> and password file <none>
I20260809 22:54:30.821627 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:30.821815 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:30.822088 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:30.823220 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786316013627029-5542-0/minicluster-data/ts-0-root/instance:
uuid: "8e330e27eed94abaaeea8f6c9e9d8b80"
format_stamp: "Formatted at 2026-08-09 22:54:30 on dist-test-slave-gp6n"
I20260809 22:54:30.828014 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 22:54:30.831611 9288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:30.832547 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:54:30.832891 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "8e330e27eed94abaaeea8f6c9e9d8b80"
format_stamp: "Formatted at 2026-08-09 22:54:30 on dist-test-slave-gp6n"
I20260809 22:54:30.833245 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:30.857924 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:30.859378 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:30.871518 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:30.871879 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:30.872221 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:30.872459 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:30.944936 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:42155
I20260809 22:54:30.944993 9369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:42155 every 8 connection(s)
I20260809 22:54:30.950937 9370 heartbeater.cc:347] Connected to a master server at 127.5.105.190:37671
I20260809 22:54:30.951320 9370 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:30.952109 9370 heartbeater.cc:511] Master 127.5.105.190:37671 requested a full tablet report, sending...
I20260809 22:54:30.954255 9210 ts_manager.cc:194] Registered new tserver with Master: 8e330e27eed94abaaeea8f6c9e9d8b80 (127.5.105.129:42155)
I20260809 22:54:30.954588 5542 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004931202s
I20260809 22:54:30.956936 9210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53276
I20260809 22:54:30.994665 9329 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260809 22:54:30.996215 9329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:31.012902 9381 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80: Bootstrap starting.
I20260809 22:54:31.017510 9381 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:31.021942 9381 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80: No bootstrap required, opened a new log
I20260809 22:54:31.022354 9381 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260809 22:54:31.024384 9381 raft_consensus.cc:348] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "8e330e27eed94abaaeea8f6c9e9d8b80" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 42155 } }
I20260809 22:54:31.024772 9381 raft_consensus.cc:374] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:31.025079 9381 raft_consensus.cc:729] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e330e27eed94abaaeea8f6c9e9d8b80, State: Initialized, Role: FOLLOWER
I20260809 22:54:31.025655 9381 consensus_queue.cc:260] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "8e330e27eed94abaaeea8f6c9e9d8b80" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 42155 } }
I20260809 22:54:31.026134 9381 raft_consensus.cc:388] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:54:31.026366 9381 raft_consensus.cc:482] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:54:31.026675 9381 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:31.031467 9381 raft_consensus.cc:504] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "8e330e27eed94abaaeea8f6c9e9d8b80" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 42155 } }
I20260809 22:54:31.032028 9381 leader_election.cc:302] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e330e27eed94abaaeea8f6c9e9d8b80; no voters:
I20260809 22:54:31.033644 9381 leader_election.cc:288] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:54:31.034104 9383 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:31.035984 9370 heartbeater.cc:503] Master 127.5.105.190:37671 was elected leader, sending a full tablet report...
I20260809 22:54:31.036623 9381 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260809 22:54:31.037230 9383 raft_consensus.cc:686] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [term 1 LEADER]: Becoming Leader. State: Replica: 8e330e27eed94abaaeea8f6c9e9d8b80, State: Running, Role: LEADER
I20260809 22:54:31.038087 9383 consensus_queue.cc:237] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "8e330e27eed94abaaeea8f6c9e9d8b80" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 42155 } }
I20260809 22:54:31.041162 9384 tablet_replica.cc:439] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e330e27eed94abaaeea8f6c9e9d8b80" committed_config { opid_index: -1 peers { permanent_uuid: "8e330e27eed94abaaeea8f6c9e9d8b80" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 42155 } } }
I20260809 22:54:31.041688 9384 tablet_replica.cc:442] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80: This TxnStatusTablet replica's current role is: LEADER
W20260809 22:54:31.043651 9210 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260809 22:54:31.043838 9385 tablet_replica.cc:439] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80: TxnStatusTablet state changed. Reason: New leader 8e330e27eed94abaaeea8f6c9e9d8b80. Latest consensus state: current_term: 1 leader_uuid: "8e330e27eed94abaaeea8f6c9e9d8b80" committed_config { opid_index: -1 peers { permanent_uuid: "8e330e27eed94abaaeea8f6c9e9d8b80" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 42155 } } }
I20260809 22:54:31.044597 9385 tablet_replica.cc:442] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:54:31.048215 9387 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 22:54:31.048722 9387 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260809 22:54:31.052536 9210 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260809 22:54:31.105559 9325 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260809 22:54:31.105930 9325 txn_status_manager.cc:243] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260809 22:54:31.106310 9325 txn_status_manager.cc:205] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20260809 22:54:31.116349 9325 txn_status_manager.cc:336] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260809 22:54:31.160302 9325 txn_status_manager.cc:243] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260809 22:54:31.160614 9325 txn_status_manager.cc:205] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20260809 22:54:31.168269 9325 txn_status_manager.cc:336] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20260809 22:54:31.183765 9329 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:53744
W20260809 22:54:31.186069 9329 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:53744
W20260809 22:54:31.188241 9329 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:53744
W20260809 22:54:31.190310 9329 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:53744
W20260809 22:54:31.192569 9329 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:53744
W20260809 22:54:31.194671 9329 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:53744
W20260809 22:54:31.207005 9329 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:53750
W20260809 22:54:31.209113 9329 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:53750
W20260809 22:54:31.211433 9329 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:53750
W20260809 22:54:31.213454 9329 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:53750
W20260809 22:54:31.215521 9329 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:53750
W20260809 22:54:31.217603 9329 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:53750
I20260809 22:54:31.223269 5542 tablet_server.cc:178] TabletServer@127.5.105.129:0 shutting down...
I20260809 22:54:31.263635 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:31.264439 5542 tablet_replica.cc:330] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80: stopping tablet replica
I20260809 22:54:31.265013 5542 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:31.265517 5542 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 8e330e27eed94abaaeea8f6c9e9d8b80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:31.279132 5542 tablet_server.cc:194] TabletServer@127.5.105.129:0 shutdown complete.
I20260809 22:54:31.292507 5542 master.cc:537] Master@127.5.105.190:37671 shutting down...
I20260809 22:54:31.310025 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:31.310560 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:31.310830 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P fda9adec3cba4c278e24bc01a4625b49: stopping tablet replica
I20260809 22:54:31.328177 5542 master.cc:552] Master@127.5.105.190:37671 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (837 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20260809 22:54:31.350018 5542 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.5.105.190:41925
I20260809 22:54:31.351115 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:31.356169 9397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:31.357302 9398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:31.359259 9401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:31.360155 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:31.360863 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:31.361060 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:31.361212 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316071361200 us; error 0 us; skew 500 ppm
I20260809 22:54:31.361721 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:31.364190 5542 webserver.cc:460] Webserver started at http://127.5.105.190:32769/ using document root <none> and password file <none>
I20260809 22:54:31.364667 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:31.364861 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:31.365124 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:31.366176 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786316013627029-5542-0/minicluster-data/master-0-root/instance:
uuid: "b1f5d636b3d448f2bb706637bc04b141"
format_stamp: "Formatted at 2026-08-09 22:54:31 on dist-test-slave-gp6n"
I20260809 22:54:31.370355 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 22:54:31.373523 9407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:31.374224 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 22:54:31.374516 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786316013627029-5542-0/minicluster-data/master-0-root
uuid: "b1f5d636b3d448f2bb706637bc04b141"
format_stamp: "Formatted at 2026-08-09 22:54:31 on dist-test-slave-gp6n"
I20260809 22:54:31.374764 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786316013627029-5542-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786316013627029-5542-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786316013627029-5542-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:31.395119 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:31.396130 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:31.432814 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.190:41925
I20260809 22:54:31.432956 9462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.190:41925 every 8 connection(s)
I20260809 22:54:31.436569 9463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:31.447410 9463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141: Bootstrap starting.
I20260809 22:54:31.452454 9463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:31.456540 9463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141: No bootstrap required, opened a new log
I20260809 22:54:31.458324 9463 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5d636b3d448f2bb706637bc04b141" member_type: VOTER }
I20260809 22:54:31.458770 9463 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:31.458993 9463 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f5d636b3d448f2bb706637bc04b141, State: Initialized, Role: FOLLOWER
I20260809 22:54:31.459523 9463 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5d636b3d448f2bb706637bc04b141" member_type: VOTER }
I20260809 22:54:31.459983 9463 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:54:31.460171 9463 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:54:31.460386 9463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:31.464525 9463 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5d636b3d448f2bb706637bc04b141" member_type: VOTER }
I20260809 22:54:31.464975 9463 leader_election.cc:302] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1f5d636b3d448f2bb706637bc04b141; no voters:
I20260809 22:54:31.466027 9463 leader_election.cc:288] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:54:31.466365 9467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:31.467725 9467 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [term 1 LEADER]: Becoming Leader. State: Replica: b1f5d636b3d448f2bb706637bc04b141, State: Running, Role: LEADER
I20260809 22:54:31.468367 9467 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5d636b3d448f2bb706637bc04b141" member_type: VOTER }
I20260809 22:54:31.469013 9463 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:54:31.471695 9470 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f5d636b3d448f2bb706637bc04b141. Latest consensus state: current_term: 1 leader_uuid: "b1f5d636b3d448f2bb706637bc04b141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5d636b3d448f2bb706637bc04b141" member_type: VOTER } }
I20260809 22:54:31.471608 9468 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1f5d636b3d448f2bb706637bc04b141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f5d636b3d448f2bb706637bc04b141" member_type: VOTER } }
I20260809 22:54:31.472395 9470 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:31.472482 9468 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:31.485306 9474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:54:31.492779 9474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:54:31.493763 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 22:54:31.501590 9474 catalog_manager.cc:1347] Generated new cluster ID: ef808fa9555642f3ac5f20b575ecb65b
I20260809 22:54:31.501873 9474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:54:31.514935 9474 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:54:31.516654 9474 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:54:31.527453 9474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141: Generated new TSK 0
I20260809 22:54:31.528224 9474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:54:31.560909 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:31.566922 9487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:31.568107 9488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:31.571573 9491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:31.573092 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:31.579229 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:31.579460 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:31.579648 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316071579630 us; error 0 us; skew 500 ppm
I20260809 22:54:31.580205 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:31.582738 5542 webserver.cc:460] Webserver started at http://127.5.105.129:43499/ using document root <none> and password file <none>
I20260809 22:54:31.583235 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:31.583437 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:31.583719 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:31.584951 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786316013627029-5542-0/minicluster-data/ts-0-root/instance:
uuid: "fa32ceb10acc416fb2531e5c9f4f1239"
format_stamp: "Formatted at 2026-08-09 22:54:31 on dist-test-slave-gp6n"
I20260809 22:54:31.589803 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260809 22:54:31.593214 9498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:31.594101 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 22:54:31.594635 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "fa32ceb10acc416fb2531e5c9f4f1239"
format_stamp: "Formatted at 2026-08-09 22:54:31 on dist-test-slave-gp6n"
I20260809 22:54:31.595194 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:31.609890 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:31.610968 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:31.616484 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:31.616950 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260809 22:54:31.617364 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:31.617705 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:31.670670 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:38893
I20260809 22:54:31.670804 9576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:38893 every 8 connection(s)
I20260809 22:54:31.675313 9577 heartbeater.cc:347] Connected to a master server at 127.5.105.190:41925
I20260809 22:54:31.675737 9577 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:31.676493 9577 heartbeater.cc:511] Master 127.5.105.190:41925 requested a full tablet report, sending...
I20260809 22:54:31.678580 9425 ts_manager.cc:194] Registered new tserver with Master: fa32ceb10acc416fb2531e5c9f4f1239 (127.5.105.129:38893)
I20260809 22:54:31.678833 5542 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.00500977s
I20260809 22:54:31.681135 9425 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39208
I20260809 22:54:31.715484 9538 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260809 22:54:31.716673 9538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:31.728019 9591 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239: Bootstrap starting.
I20260809 22:54:31.733361 9591 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:31.737672 9591 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239: No bootstrap required, opened a new log
I20260809 22:54:31.738111 9591 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 22:54:31.739975 9591 raft_consensus.cc:348] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "fa32ceb10acc416fb2531e5c9f4f1239" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38893 } }
I20260809 22:54:31.740396 9591 raft_consensus.cc:374] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:31.740639 9591 raft_consensus.cc:729] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa32ceb10acc416fb2531e5c9f4f1239, State: Initialized, Role: FOLLOWER
I20260809 22:54:31.741214 9591 consensus_queue.cc:260] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "fa32ceb10acc416fb2531e5c9f4f1239" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38893 } }
I20260809 22:54:31.741705 9591 raft_consensus.cc:388] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:54:31.741948 9591 raft_consensus.cc:482] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:54:31.742208 9591 raft_consensus.cc:3037] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:31.746657 9591 raft_consensus.cc:504] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "fa32ceb10acc416fb2531e5c9f4f1239" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38893 } }
I20260809 22:54:31.747184 9591 leader_election.cc:302] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa32ceb10acc416fb2531e5c9f4f1239; no voters:
I20260809 22:54:31.748317 9591 leader_election.cc:288] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:54:31.748710 9593 raft_consensus.cc:2781] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:31.750100 9593 raft_consensus.cc:686] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [term 1 LEADER]: Becoming Leader. State: Replica: fa32ceb10acc416fb2531e5c9f4f1239, State: Running, Role: LEADER
I20260809 22:54:31.750319 9577 heartbeater.cc:503] Master 127.5.105.190:41925 was elected leader, sending a full tablet report...
I20260809 22:54:31.750792 9591 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239: Time spent starting tablet: real 0.012s user 0.010s sys 0.004s
I20260809 22:54:31.750916 9593 consensus_queue.cc:237] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "fa32ceb10acc416fb2531e5c9f4f1239" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38893 } }
I20260809 22:54:31.754886 9594 tablet_replica.cc:439] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa32ceb10acc416fb2531e5c9f4f1239" committed_config { opid_index: -1 peers { permanent_uuid: "fa32ceb10acc416fb2531e5c9f4f1239" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38893 } } }
I20260809 22:54:31.755468 9594 tablet_replica.cc:442] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:54:31.756142 9596 tablet_replica.cc:439] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239: TxnStatusTablet state changed. Reason: New leader fa32ceb10acc416fb2531e5c9f4f1239. Latest consensus state: current_term: 1 leader_uuid: "fa32ceb10acc416fb2531e5c9f4f1239" committed_config { opid_index: -1 peers { permanent_uuid: "fa32ceb10acc416fb2531e5c9f4f1239" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38893 } } }
I20260809 22:54:31.756799 9596 tablet_replica.cc:442] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:54:31.758872 9599 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
W20260809 22:54:31.759059 9425 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260809 22:54:31.759337 9599 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260809 22:54:31.763172 9425 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260809 22:54:31.788687 9538 tablet_service.cc:1467] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260809 22:54:31.789938 9538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:31.800110 9591 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239: Bootstrap starting.
I20260809 22:54:31.804441 9591 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:31.808471 9591 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239: No bootstrap required, opened a new log
I20260809 22:54:31.808893 9591 ts_tablet_manager.cc:1397] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 22:54:31.810822 9591 raft_consensus.cc:348] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "fa32ceb10acc416fb2531e5c9f4f1239" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38893 } }
I20260809 22:54:31.811240 9591 raft_consensus.cc:374] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:31.811462 9591 raft_consensus.cc:729] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa32ceb10acc416fb2531e5c9f4f1239, State: Initialized, Role: FOLLOWER
I20260809 22:54:31.811995 9591 consensus_queue.cc:260] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "fa32ceb10acc416fb2531e5c9f4f1239" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38893 } }
I20260809 22:54:31.812453 9591 raft_consensus.cc:388] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:54:31.812688 9591 raft_consensus.cc:482] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:54:31.812933 9591 raft_consensus.cc:3037] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:31.817318 9591 raft_consensus.cc:504] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "fa32ceb10acc416fb2531e5c9f4f1239" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38893 } }
I20260809 22:54:31.817857 9591 leader_election.cc:302] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa32ceb10acc416fb2531e5c9f4f1239; no voters:
I20260809 22:54:31.818343 9591 leader_election.cc:288] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:54:31.818638 9593 raft_consensus.cc:2781] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:31.819171 9593 raft_consensus.cc:686] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [term 1 LEADER]: Becoming Leader. State: Replica: fa32ceb10acc416fb2531e5c9f4f1239, State: Running, Role: LEADER
I20260809 22:54:31.819828 9591 ts_tablet_manager.cc:1428] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260809 22:54:31.819839 9593 consensus_queue.cc:237] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "fa32ceb10acc416fb2531e5c9f4f1239" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38893 } }
W20260809 22:54:31.825610 9425 catalog_manager.cc:5328] Ignoring report from unknown tablet 22222222222222222222222222222222
W20260809 22:54:31.829721 9425 catalog_manager.cc:5328] Ignoring report from unknown tablet 22222222222222222222222222222222
I20260809 22:54:31.850554 5542 tablet_server.cc:178] TabletServer@127.5.105.129:0 shutting down...
I20260809 22:54:31.875756 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:31.876646 5542 tablet_replica.cc:330] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239: stopping tablet replica
I20260809 22:54:31.877679 5542 raft_consensus.cc:2229] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:31.878114 5542 raft_consensus.cc:2258] T 22222222222222222222222222222222 P fa32ceb10acc416fb2531e5c9f4f1239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:31.880558 5542 tablet_replica.cc:330] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239: stopping tablet replica
I20260809 22:54:31.881023 5542 raft_consensus.cc:2229] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:31.881457 5542 raft_consensus.cc:2258] T 11111111111111111111111111111111 P fa32ceb10acc416fb2531e5c9f4f1239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:31.903141 5542 tablet_server.cc:194] TabletServer@127.5.105.129:0 shutdown complete.
I20260809 22:54:31.914780 5542 master.cc:537] Master@127.5.105.190:41925 shutting down...
I20260809 22:54:31.930132 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:31.930720 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:31.931207 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P b1f5d636b3d448f2bb706637bc04b141: stopping tablet replica
I20260809 22:54:31.950139 5542 master.cc:552] Master@127.5.105.190:41925 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (623 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (3566 ms total)
[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/0
I20260809 22:54:31.974002 5542 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.5.105.190:45807
I20260809 22:54:31.975026 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:31.979745 9610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:31.980983 9611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:31.982318 9613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:31.982554 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:31.983327 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:31.983559 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:31.983666 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316071983658 us; error 0 us; skew 500 ppm
I20260809 22:54:31.984118 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:31.986342 5542 webserver.cc:460] Webserver started at http://127.5.105.190:37355/ using document root <none> and password file <none>
I20260809 22:54:31.986819 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:31.986960 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:31.987154 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:31.988197 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/master-0-root/instance:
uuid: "7c95ed659ad44368ad3bcb9438fafad0"
format_stamp: "Formatted at 2026-08-09 22:54:31 on dist-test-slave-gp6n"
I20260809 22:54:31.992632 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 22:54:31.996026 9618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:31.996805 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 22:54:31.997083 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/master-0-root
uuid: "7c95ed659ad44368ad3bcb9438fafad0"
format_stamp: "Formatted at 2026-08-09 22:54:31 on dist-test-slave-gp6n"
I20260809 22:54:31.997336 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:32.017131 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:32.018205 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:32.059393 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.190:45807
I20260809 22:54:32.059527 9676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.190:45807 every 8 connection(s)
I20260809 22:54:32.063109 9677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:32.073154 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0: Bootstrap starting.
I20260809 22:54:32.077117 9677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:32.081014 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0: No bootstrap required, opened a new log
I20260809 22:54:32.082882 9677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c95ed659ad44368ad3bcb9438fafad0" member_type: VOTER }
I20260809 22:54:32.083233 9677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:32.083508 9677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c95ed659ad44368ad3bcb9438fafad0, State: Initialized, Role: FOLLOWER
I20260809 22:54:32.084126 9677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c95ed659ad44368ad3bcb9438fafad0" member_type: VOTER }
I20260809 22:54:32.084573 9677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:54:32.084780 9677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:54:32.085039 9677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:32.089692 9677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c95ed659ad44368ad3bcb9438fafad0" member_type: VOTER }
I20260809 22:54:32.090197 9677 leader_election.cc:302] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c95ed659ad44368ad3bcb9438fafad0; no voters:
I20260809 22:54:32.091442 9677 leader_election.cc:288] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:54:32.091775 9680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:32.093118 9680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [term 1 LEADER]: Becoming Leader. State: Replica: 7c95ed659ad44368ad3bcb9438fafad0, State: Running, Role: LEADER
I20260809 22:54:32.093751 9680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c95ed659ad44368ad3bcb9438fafad0" member_type: VOTER }
I20260809 22:54:32.094466 9677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:54:32.096711 9682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c95ed659ad44368ad3bcb9438fafad0. Latest consensus state: current_term: 1 leader_uuid: "7c95ed659ad44368ad3bcb9438fafad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c95ed659ad44368ad3bcb9438fafad0" member_type: VOTER } }
I20260809 22:54:32.096683 9681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c95ed659ad44368ad3bcb9438fafad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c95ed659ad44368ad3bcb9438fafad0" member_type: VOTER } }
I20260809 22:54:32.097419 9682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:32.097795 9681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:32.101217 9685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:54:32.105773 9685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:54:32.109522 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 22:54:32.113866 9685 catalog_manager.cc:1347] Generated new cluster ID: c74a187b8b634d729911e31b9d251962
I20260809 22:54:32.114109 9685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:54:32.140560 9685 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:54:32.142439 9685 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:54:32.156589 9685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0: Generated new TSK 0
I20260809 22:54:32.157222 9685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:54:32.175733 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:32.181612 9700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:32.182617 9701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:32.184911 5542 server_base.cc:1034] running on GCE node
W20260809 22:54:32.184963 9703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:32.185957 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:32.186138 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:32.186308 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316072186297 us; error 0 us; skew 500 ppm
I20260809 22:54:32.186836 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:32.189173 5542 webserver.cc:460] Webserver started at http://127.5.105.129:37935/ using document root <none> and password file <none>
I20260809 22:54:32.189610 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:32.189755 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:32.189962 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:32.191093 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-0-root/instance:
uuid: "df5fcb65a69c43ebbeea4b62827bfb26"
format_stamp: "Formatted at 2026-08-09 22:54:32 on dist-test-slave-gp6n"
I20260809 22:54:32.195421 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260809 22:54:32.198805 9708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:32.199596 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260809 22:54:32.199862 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "df5fcb65a69c43ebbeea4b62827bfb26"
format_stamp: "Formatted at 2026-08-09 22:54:32 on dist-test-slave-gp6n"
I20260809 22:54:32.200089 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:32.210657 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:32.211536 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:32.212836 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:54:32.215095 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:32.215258 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:32.215529 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:32.215694 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:32.256979 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:39737
I20260809 22:54:32.257140 9775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:39737 every 8 connection(s)
I20260809 22:54:32.261907 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:32.269212 9780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:32.269397 9781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:32.272621 5542 server_base.cc:1034] running on GCE node
W20260809 22:54:32.273744 9783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:32.274847 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:32.275087 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:32.275279 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316072275264 us; error 0 us; skew 500 ppm
I20260809 22:54:32.276006 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:32.279431 9776 heartbeater.cc:347] Connected to a master server at 127.5.105.190:45807
I20260809 22:54:32.279481 5542 webserver.cc:460] Webserver started at http://127.5.105.130:42227/ using document root <none> and password file <none>
I20260809 22:54:32.279920 9776 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:32.280287 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:32.280598 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:32.280647 9776 heartbeater.cc:511] Master 127.5.105.190:45807 requested a full tablet report, sending...
I20260809 22:54:32.281076 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:32.282604 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-1-root/instance:
uuid: "329018d9fc43482486d43ea2cacc7bec"
format_stamp: "Formatted at 2026-08-09 22:54:32 on dist-test-slave-gp6n"
I20260809 22:54:32.282846 9635 ts_manager.cc:194] Registered new tserver with Master: df5fcb65a69c43ebbeea4b62827bfb26 (127.5.105.129:39737)
I20260809 22:54:32.284691 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59686
I20260809 22:54:32.287745 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260809 22:54:32.291769 9790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:32.292555 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260809 22:54:32.292865 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-1-root
uuid: "329018d9fc43482486d43ea2cacc7bec"
format_stamp: "Formatted at 2026-08-09 22:54:32 on dist-test-slave-gp6n"
I20260809 22:54:32.293160 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:32.314813 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:32.315924 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:32.317484 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:54:32.319952 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:32.320166 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:32.320432 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:32.320595 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:32.362151 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.130:36671
I20260809 22:54:32.362303 9860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.130:36671 every 8 connection(s)
I20260809 22:54:32.367391 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:32.374585 9864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:32.375192 9865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:32.380723 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:32.381819 9861 heartbeater.cc:347] Connected to a master server at 127.5.105.190:45807
W20260809 22:54:32.381909 9867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:32.382292 9861 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:32.383049 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260809 22:54:32.383131 9861 heartbeater.cc:511] Master 127.5.105.190:45807 requested a full tablet report, sending...
W20260809 22:54:32.383442 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:32.383701 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316072383684 us; error 0 us; skew 500 ppm
I20260809 22:54:32.384490 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:32.385715 9635 ts_manager.cc:194] Registered new tserver with Master: 329018d9fc43482486d43ea2cacc7bec (127.5.105.130:36671)
I20260809 22:54:32.387629 5542 webserver.cc:460] Webserver started at http://127.5.105.131:33823/ using document root <none> and password file <none>
I20260809 22:54:32.387768 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59688
I20260809 22:54:32.388236 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:32.388489 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:32.388829 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:32.390050 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-2-root/instance:
uuid: "2d453524404b4643a1a2b6df91f0d19a"
format_stamp: "Formatted at 2026-08-09 22:54:32 on dist-test-slave-gp6n"
I20260809 22:54:32.395061 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 22:54:32.398658 9872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:32.399530 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 22:54:32.399835 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-2-root
uuid: "2d453524404b4643a1a2b6df91f0d19a"
format_stamp: "Formatted at 2026-08-09 22:54:32 on dist-test-slave-gp6n"
I20260809 22:54:32.400157 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:32.410149 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:32.411207 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:32.412554 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:54:32.414758 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:32.414975 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:32.415201 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:32.415374 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:32.456650 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.131:33235
I20260809 22:54:32.456782 9946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.131:33235 every 8 connection(s)
I20260809 22:54:32.473420 9947 heartbeater.cc:347] Connected to a master server at 127.5.105.190:45807
I20260809 22:54:32.473801 9947 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:32.474530 9947 heartbeater.cc:511] Master 127.5.105.190:45807 requested a full tablet report, sending...
I20260809 22:54:32.476464 9635 ts_manager.cc:194] Registered new tserver with Master: 2d453524404b4643a1a2b6df91f0d19a (127.5.105.131:33235)
I20260809 22:54:32.477017 5542 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017094629s
I20260809 22:54:32.478189 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59698
I20260809 22:54:32.500627 5542 test_util.cc:268] Using random seed: 519388521
I20260809 22:54:32.525403 9635 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59718:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 22:54:32.575201 9906 tablet_service.cc:1467] Processing CreateTablet for tablet 3b41d5fe85cf4ee3abf3f871f42c687b (DEFAULT_TABLE table=test-workload [id=e976ce826e4f42e8a0946defc111d94e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:54:32.576472 9906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b41d5fe85cf4ee3abf3f871f42c687b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:32.578997 9822 tablet_service.cc:1467] Processing CreateTablet for tablet 3b41d5fe85cf4ee3abf3f871f42c687b (DEFAULT_TABLE table=test-workload [id=e976ce826e4f42e8a0946defc111d94e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:54:32.580380 9822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b41d5fe85cf4ee3abf3f871f42c687b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:32.583963 9738 tablet_service.cc:1467] Processing CreateTablet for tablet 3b41d5fe85cf4ee3abf3f871f42c687b (DEFAULT_TABLE table=test-workload [id=e976ce826e4f42e8a0946defc111d94e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:54:32.585292 9738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b41d5fe85cf4ee3abf3f871f42c687b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:32.595702 9969 tablet_bootstrap.cc:492] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a: Bootstrap starting.
I20260809 22:54:32.603919 9970 tablet_bootstrap.cc:492] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: Bootstrap starting.
I20260809 22:54:32.604511 9969 tablet_bootstrap.cc:654] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:32.608754 9970 tablet_bootstrap.cc:654] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:32.609171 9971 tablet_bootstrap.cc:492] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26: Bootstrap starting.
I20260809 22:54:32.613469 9970 tablet_bootstrap.cc:492] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: No bootstrap required, opened a new log
I20260809 22:54:32.613866 9970 ts_tablet_manager.cc:1397] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 22:54:32.614396 9969 tablet_bootstrap.cc:492] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a: No bootstrap required, opened a new log
I20260809 22:54:32.614917 9969 ts_tablet_manager.cc:1397] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.000s
I20260809 22:54:32.615398 9971 tablet_bootstrap.cc:654] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:32.616137 9970 raft_consensus.cc:348] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } }
I20260809 22:54:32.616719 9970 raft_consensus.cc:374] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:32.616976 9970 raft_consensus.cc:729] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 329018d9fc43482486d43ea2cacc7bec, State: Initialized, Role: FOLLOWER
I20260809 22:54:32.617592 9970 consensus_queue.cc:260] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } }
I20260809 22:54:32.617887 9969 raft_consensus.cc:348] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } }
I20260809 22:54:32.618872 9969 raft_consensus.cc:374] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:32.619210 9969 raft_consensus.cc:729] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d453524404b4643a1a2b6df91f0d19a, State: Initialized, Role: FOLLOWER
I20260809 22:54:32.619994 9969 consensus_queue.cc:260] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } }
I20260809 22:54:32.620795 9971 tablet_bootstrap.cc:492] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26: No bootstrap required, opened a new log
I20260809 22:54:32.621308 9971 ts_tablet_manager.cc:1397] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260809 22:54:32.623555 9971 raft_consensus.cc:348] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } }
I20260809 22:54:32.624131 9971 raft_consensus.cc:374] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:32.624460 9971 raft_consensus.cc:729] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5fcb65a69c43ebbeea4b62827bfb26, State: Initialized, Role: FOLLOWER
I20260809 22:54:32.625710 9970 ts_tablet_manager.cc:1428] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260809 22:54:32.625355 9971 consensus_queue.cc:260] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } }
I20260809 22:54:32.626171 9861 heartbeater.cc:503] Master 127.5.105.190:45807 was elected leader, sending a full tablet report...
I20260809 22:54:32.628294 9947 heartbeater.cc:503] Master 127.5.105.190:45807 was elected leader, sending a full tablet report...
I20260809 22:54:32.628842 9969 ts_tablet_manager.cc:1428] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20260809 22:54:32.630471 9776 heartbeater.cc:503] Master 127.5.105.190:45807 was elected leader, sending a full tablet report...
I20260809 22:54:32.631207 9971 ts_tablet_manager.cc:1428] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260809 22:54:32.728726 9976 raft_consensus.cc:482] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:54:32.729297 9976 raft_consensus.cc:504] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } }
I20260809 22:54:32.731228 9976 leader_election.cc:288] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 329018d9fc43482486d43ea2cacc7bec (127.5.105.130:36671), df5fcb65a69c43ebbeea4b62827bfb26 (127.5.105.129:39737)
I20260809 22:54:32.745623 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b41d5fe85cf4ee3abf3f871f42c687b" candidate_uuid: "2d453524404b4643a1a2b6df91f0d19a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "329018d9fc43482486d43ea2cacc7bec" is_pre_election: true
I20260809 22:54:32.746503 9833 raft_consensus.cc:2454] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d453524404b4643a1a2b6df91f0d19a in term 0.
I20260809 22:54:32.747620 9874 leader_election.cc:302] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d453524404b4643a1a2b6df91f0d19a, 329018d9fc43482486d43ea2cacc7bec; no voters:
I20260809 22:54:32.748471 9976 raft_consensus.cc:2781] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:54:32.748214 9749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b41d5fe85cf4ee3abf3f871f42c687b" candidate_uuid: "2d453524404b4643a1a2b6df91f0d19a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" is_pre_election: true
I20260809 22:54:32.748888 9976 raft_consensus.cc:482] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:54:32.749140 9749 raft_consensus.cc:2454] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d453524404b4643a1a2b6df91f0d19a in term 0.
I20260809 22:54:32.749264 9976 raft_consensus.cc:3037] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:32.754155 9976 raft_consensus.cc:504] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } }
I20260809 22:54:32.755563 9976 leader_election.cc:288] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [CANDIDATE]: Term 1 election: Requested vote from peers 329018d9fc43482486d43ea2cacc7bec (127.5.105.130:36671), df5fcb65a69c43ebbeea4b62827bfb26 (127.5.105.129:39737)
I20260809 22:54:32.756381 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b41d5fe85cf4ee3abf3f871f42c687b" candidate_uuid: "2d453524404b4643a1a2b6df91f0d19a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "329018d9fc43482486d43ea2cacc7bec"
I20260809 22:54:32.756618 9749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b41d5fe85cf4ee3abf3f871f42c687b" candidate_uuid: "2d453524404b4643a1a2b6df91f0d19a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5fcb65a69c43ebbeea4b62827bfb26"
I20260809 22:54:32.756970 9833 raft_consensus.cc:3037] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:32.757148 9749 raft_consensus.cc:3037] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:32.763435 9833 raft_consensus.cc:2454] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d453524404b4643a1a2b6df91f0d19a in term 1.
I20260809 22:54:32.763435 9749 raft_consensus.cc:2454] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d453524404b4643a1a2b6df91f0d19a in term 1.
I20260809 22:54:32.764549 9874 leader_election.cc:302] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d453524404b4643a1a2b6df91f0d19a, df5fcb65a69c43ebbeea4b62827bfb26; no voters:
I20260809 22:54:32.765102 9976 raft_consensus.cc:2781] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:32.765938 9976 raft_consensus.cc:686] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 1 LEADER]: Becoming Leader. State: Replica: 2d453524404b4643a1a2b6df91f0d19a, State: Running, Role: LEADER
I20260809 22:54:32.766745 9976 consensus_queue.cc:237] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } }
I20260809 22:54:32.773365 9634 catalog_manager.cc:5495] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d453524404b4643a1a2b6df91f0d19a (127.5.105.131). New cstate: current_term: 1 leader_uuid: "2d453524404b4643a1a2b6df91f0d19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } }
I20260809 22:54:33.061546 9906 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260809 22:54:33.065788 9822 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:33.070428 9738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:33.236466 9976 consensus_queue.cc:1035] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [LEADER]: Connected to new peer: Peer: permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:54:33.255302 9980 consensus_queue.cc:1035] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
7c95ed659ad44368ad3bcb9438fafad0 | 127.5.105.190:45807 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2d453524404b4643a1a2b6df91f0d19a | 127.5.105.131:33235 | HEALTHY | <none> | 1 | 0
329018d9fc43482486d43ea2cacc7bec | 127.5.105.130:36671 | HEALTHY | <none> | 0 | 0
df5fcb65a69c43ebbeea4b62827bfb26 | 127.5.105.129:39737 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 3b41d5fe85cf4ee3abf3f871f42c687b of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
2d453524404b4643a1a2b6df91f0d19a (127.5.105.131:33235): RUNNING [LEADER]
329018d9fc43482486d43ea2cacc7bec (127.5.105.130:36671): RUNNING
df5fcb65a69c43ebbeea4b62827bfb26 (127.5.105.129:39737): RUNNING
All reported replicas are:
A = 2d453524404b4643a1a2b6df91f0d19a
B = 329018d9fc43482486d43ea2cacc7bec
C = df5fcb65a69c43ebbeea4b62827bfb26
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 22:54:33.302711 5542 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 22:54:33.663861 9906 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260809 22:54:33.670961 9822 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:33.672194 9738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
7c95ed659ad44368ad3bcb9438fafad0 | 127.5.105.190:45807 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2d453524404b4643a1a2b6df91f0d19a | 127.5.105.131:33235 | HEALTHY | <none> | 1 | 0
329018d9fc43482486d43ea2cacc7bec | 127.5.105.130:36671 | HEALTHY | <none> | 0 | 0
df5fcb65a69c43ebbeea4b62827bfb26 | 127.5.105.129:39737 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260809 22:54:33.847820 5542 log_verifier.cc:126] Checking tablet 3b41d5fe85cf4ee3abf3f871f42c687b
I20260809 22:54:33.912401 5542 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 3b41d5fe85cf4ee3abf3f871f42c687b
I20260809 22:54:33.912993 5542 tablet_replica.cc:330] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: stopping tablet replica
I20260809 22:54:33.913527 5542 raft_consensus.cc:2229] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:33.913988 5542 raft_consensus.cc:2258] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:34.130271 9906 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260809 22:54:34.130746 9822 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:34.132064 9738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20260809 22:54:34.305531 9874 consensus_peers.cc:487] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a -> Peer 329018d9fc43482486d43ea2cacc7bec (127.5.105.130:36671): Couldn't send request to peer 329018d9fc43482486d43ea2cacc7bec. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260809 22:54:34.319060 10079 raft_consensus.cc:998] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a: Attempting to remove follower 329018d9fc43482486d43ea2cacc7bec from the Raft config. Reason: The tablet replica hosted on peer 329018d9fc43482486d43ea2cacc7bec has failed
I20260809 22:54:34.323230 10079 consensus_queue.cc:237] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } }
I20260809 22:54:34.331209 9749 raft_consensus.cc:1261] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 1 FOLLOWER]: Refusing update from remote peer 2d453524404b4643a1a2b6df91f0d19a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:54:34.333241 10079 consensus_queue.cc:1035] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 22:54:34.353714 9980 raft_consensus.cc:2932] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 329018d9fc43482486d43ea2cacc7bec (127.5.105.130) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } }
I20260809 22:54:34.365957 9749 raft_consensus.cc:2932] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 329018d9fc43482486d43ea2cacc7bec (127.5.105.130) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } }
I20260809 22:54:34.372563 9633 catalog_manager.cc:5495] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a reported cstate change: config changed from index -1 to 2, VOTER 329018d9fc43482486d43ea2cacc7bec (127.5.105.130) evicted. New cstate: current_term: 1 leader_uuid: "2d453524404b4643a1a2b6df91f0d19a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } }
I20260809 22:54:34.424914 9822 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b41d5fe85cf4ee3abf3f871f42c687b with delete_type TABLET_DATA_TOMBSTONED (TS 329018d9fc43482486d43ea2cacc7bec not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:56484
I20260809 22:54:34.428960 9919 consensus_queue.cc:237] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } }
W20260809 22:54:34.439433 9874 consensus_peers.cc:487] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a -> Peer 329018d9fc43482486d43ea2cacc7bec (127.5.105.130:36671): Couldn't send request to peer 329018d9fc43482486d43ea2cacc7bec. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260809 22:54:34.442734 9749 raft_consensus.cc:1261] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 1 FOLLOWER]: Refusing update from remote peer 2d453524404b4643a1a2b6df91f0d19a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 22:54:34.444126 10079 consensus_queue.cc:1035] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 22:54:34.445920 10090 ts_tablet_manager.cc:1905] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 22:54:34.450430 9980 raft_consensus.cc:2932] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 329018d9fc43482486d43ea2cacc7bec (127.5.105.130) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } }
I20260809 22:54:34.454665 9749 raft_consensus.cc:2932] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 329018d9fc43482486d43ea2cacc7bec (127.5.105.130) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } }
I20260809 22:54:34.461337 9621 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet 3b41d5fe85cf4ee3abf3f871f42c687b with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260809 22:54:34.463606 9633 catalog_manager.cc:5495] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a reported cstate change: config changed from index 2 to 3, VOTER 329018d9fc43482486d43ea2cacc7bec (127.5.105.130) added. New cstate: current_term: 1 leader_uuid: "2d453524404b4643a1a2b6df91f0d19a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } }
I20260809 22:54:34.466321 10090 ts_tablet_manager.cc:1918] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260809 22:54:34.466807 10090 log.cc:1196] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: Deleting WAL directory at /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786316013627029-5542-0/minicluster-data/ts-1-root/wals/3b41d5fe85cf4ee3abf3f871f42c687b
I20260809 22:54:34.469476 9620 catalog_manager.cc:4841] TS 329018d9fc43482486d43ea2cacc7bec (127.5.105.130:36671): tablet 3b41d5fe85cf4ee3abf3f871f42c687b (table test-workload [id=e976ce826e4f42e8a0946defc111d94e]) successfully deleted
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
7c95ed659ad44368ad3bcb9438fafad0 | 127.5.105.190:45807 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2d453524404b4643a1a2b6df91f0d19a | 127.5.105.131:33235 | HEALTHY | <none> | 1 | 0
329018d9fc43482486d43ea2cacc7bec | 127.5.105.130:36671 | HEALTHY | <none> | 0 | 0
df5fcb65a69c43ebbeea4b62827bfb26 | 127.5.105.129:39737 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 3b41d5fe85cf4ee3abf3f871f42c687b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
2d453524404b4643a1a2b6df91f0d19a (127.5.105.131:33235): RUNNING [LEADER]
329018d9fc43482486d43ea2cacc7bec (127.5.105.130:36671): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
df5fcb65a69c43ebbeea4b62827bfb26 (127.5.105.129:39737): RUNNING
All reported replicas are:
A = 2d453524404b4643a1a2b6df91f0d19a
B = 329018d9fc43482486d43ea2cacc7bec
C = df5fcb65a69c43ebbeea4b62827bfb26
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 22:54:34.486306 5542 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 22:54:34.861577 9906 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260809 22:54:34.875396 9738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:34.875753 9822 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:34.880991 9980 raft_consensus.cc:998] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a: Attempting to remove follower 329018d9fc43482486d43ea2cacc7bec from the Raft config. Reason: The tablet replica hosted on peer 329018d9fc43482486d43ea2cacc7bec has failed
I20260809 22:54:34.882869 9980 consensus_queue.cc:237] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } }
I20260809 22:54:34.886955 9749 raft_consensus.cc:1261] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 1 FOLLOWER]: Refusing update from remote peer 2d453524404b4643a1a2b6df91f0d19a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 22:54:34.890865 9980 consensus_queue.cc:1035] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 22:54:34.897357 10079 raft_consensus.cc:2932] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 329018d9fc43482486d43ea2cacc7bec (127.5.105.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } }
I20260809 22:54:34.899202 9749 raft_consensus.cc:2932] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 329018d9fc43482486d43ea2cacc7bec (127.5.105.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } }
I20260809 22:54:34.915282 9633 catalog_manager.cc:5495] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a reported cstate change: config changed from index 3 to 4, VOTER 329018d9fc43482486d43ea2cacc7bec (127.5.105.130) evicted. New cstate: current_term: 1 leader_uuid: "2d453524404b4643a1a2b6df91f0d19a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } }
I20260809 22:54:34.926330 9822 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b41d5fe85cf4ee3abf3f871f42c687b with delete_type TABLET_DATA_TOMBSTONED (TS 329018d9fc43482486d43ea2cacc7bec not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56484
I20260809 22:54:34.930596 9919 consensus_queue.cc:237] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } }
I20260809 22:54:34.932533 9620 catalog_manager.cc:4841] TS 329018d9fc43482486d43ea2cacc7bec (127.5.105.130:36671): tablet 3b41d5fe85cf4ee3abf3f871f42c687b (table test-workload [id=e976ce826e4f42e8a0946defc111d94e]) successfully deleted
I20260809 22:54:34.938812 9749 raft_consensus.cc:1261] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 1 FOLLOWER]: Refusing update from remote peer 2d453524404b4643a1a2b6df91f0d19a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260809 22:54:34.938966 9874 consensus_peers.cc:487] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a -> Peer 329018d9fc43482486d43ea2cacc7bec (127.5.105.130:36671): Couldn't send request to peer 329018d9fc43482486d43ea2cacc7bec. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260809 22:54:34.940582 9980 consensus_queue.cc:1035] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 22:54:34.947546 10079 raft_consensus.cc:2932] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 329018d9fc43482486d43ea2cacc7bec (127.5.105.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } }
I20260809 22:54:34.958321 9747 raft_consensus.cc:2932] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 329018d9fc43482486d43ea2cacc7bec (127.5.105.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } }
I20260809 22:54:34.968139 9621 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet 3b41d5fe85cf4ee3abf3f871f42c687b with cas_config_opid_index 4: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260809 22:54:34.973224 9635 catalog_manager.cc:5495] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a reported cstate change: config changed from index 4 to 5, VOTER 329018d9fc43482486d43ea2cacc7bec (127.5.105.130) added. New cstate: current_term: 1 leader_uuid: "2d453524404b4643a1a2b6df91f0d19a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
7c95ed659ad44368ad3bcb9438fafad0 | 127.5.105.190:45807 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2d453524404b4643a1a2b6df91f0d19a | 127.5.105.131:33235 | HEALTHY | <none> | 1 | 0
329018d9fc43482486d43ea2cacc7bec | 127.5.105.130:36671 | HEALTHY | <none> | 0 | 0
df5fcb65a69c43ebbeea4b62827bfb26 | 127.5.105.129:39737 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 3b41d5fe85cf4ee3abf3f871f42c687b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
2d453524404b4643a1a2b6df91f0d19a (127.5.105.131:33235): RUNNING [LEADER]
df5fcb65a69c43ebbeea4b62827bfb26 (127.5.105.129:39737): RUNNING
329018d9fc43482486d43ea2cacc7bec (127.5.105.130:36671): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 2d453524404b4643a1a2b6df91f0d19a
B = df5fcb65a69c43ebbeea4b62827bfb26
C = 329018d9fc43482486d43ea2cacc7bec
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 22:54:35.084455 5542 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 22:54:35.461676 10142 ts_tablet_manager.cc:927] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: Initiating tablet copy from peer 2d453524404b4643a1a2b6df91f0d19a (127.5.105.131:33235)
I20260809 22:54:35.463419 10142 tablet_copy_client.cc:316] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: tablet copy: Beginning tablet copy session from remote peer at address 127.5.105.131:33235
I20260809 22:54:35.483403 9929 tablet_copy_service.cc:140] P 2d453524404b4643a1a2b6df91f0d19a: Received BeginTabletCopySession request for tablet 3b41d5fe85cf4ee3abf3f871f42c687b from peer 329018d9fc43482486d43ea2cacc7bec ({username='slave'} at 127.0.0.1:53002)
I20260809 22:54:35.484056 9929 tablet_copy_service.cc:161] P 2d453524404b4643a1a2b6df91f0d19a: Beginning new tablet copy session on tablet 3b41d5fe85cf4ee3abf3f871f42c687b from peer 329018d9fc43482486d43ea2cacc7bec at {username='slave'} at 127.0.0.1:53002: session id = 329018d9fc43482486d43ea2cacc7bec-3b41d5fe85cf4ee3abf3f871f42c687b
I20260809 22:54:35.491597 9929 tablet_copy_source_session.cc:205] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 22:54:35.494148 10142 ts_tablet_manager.cc:1905] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260809 22:54:35.508934 10142 ts_tablet_manager.cc:1918] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260809 22:54:35.509667 10142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b41d5fe85cf4ee3abf3f871f42c687b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:35.518287 10142 tablet_copy_client.cc:791] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: tablet copy: Starting download of 0 data blocks...
I20260809 22:54:35.518993 10142 tablet_copy_client.cc:655] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: tablet copy: Starting download of 1 WAL segments...
I20260809 22:54:35.523550 10142 tablet_copy_client.cc:531] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 22:54:35.535516 10142 tablet_bootstrap.cc:492] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: Bootstrap starting.
I20260809 22:54:35.541958 9906 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260809 22:54:35.548768 9822 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:35.559203 9738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:35.563930 10142 tablet_bootstrap.cc:492] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 22:54:35.564826 10142 tablet_bootstrap.cc:492] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: Bootstrap complete.
I20260809 22:54:35.565430 10142 ts_tablet_manager.cc:1397] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: Time spent bootstrapping tablet: real 0.030s user 0.020s sys 0.009s
I20260809 22:54:35.567713 10142 raft_consensus.cc:348] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } }
I20260809 22:54:35.568300 10142 raft_consensus.cc:729] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 329018d9fc43482486d43ea2cacc7bec, State: Initialized, Role: FOLLOWER
I20260809 22:54:35.568974 10142 consensus_queue.cc:260] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d453524404b4643a1a2b6df91f0d19a" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 33235 } } peers { permanent_uuid: "df5fcb65a69c43ebbeea4b62827bfb26" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 39737 } } peers { permanent_uuid: "329018d9fc43482486d43ea2cacc7bec" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 36671 } }
I20260809 22:54:35.571872 10142 ts_tablet_manager.cc:1428] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 22:54:35.573935 9929 tablet_copy_service.cc:342] P 2d453524404b4643a1a2b6df91f0d19a: Request end of tablet copy session 329018d9fc43482486d43ea2cacc7bec-3b41d5fe85cf4ee3abf3f871f42c687b received from {username='slave'} at 127.0.0.1:53002
I20260809 22:54:35.574357 9929 tablet_copy_service.cc:433] P 2d453524404b4643a1a2b6df91f0d19a: ending tablet copy session 329018d9fc43482486d43ea2cacc7bec-3b41d5fe85cf4ee3abf3f871f42c687b on tablet 3b41d5fe85cf4ee3abf3f871f42c687b with peer 329018d9fc43482486d43ea2cacc7bec
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
7c95ed659ad44368ad3bcb9438fafad0 | 127.5.105.190:45807 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2d453524404b4643a1a2b6df91f0d19a | 127.5.105.131:33235 | HEALTHY | <none> | 1 | 0
329018d9fc43482486d43ea2cacc7bec | 127.5.105.130:36671 | HEALTHY | <none> | 0 | 0
df5fcb65a69c43ebbeea4b62827bfb26 | 127.5.105.129:39737 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 3b41d5fe85cf4ee3abf3f871f42c687b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
2d453524404b4643a1a2b6df91f0d19a (127.5.105.131:33235): RUNNING [LEADER]
df5fcb65a69c43ebbeea4b62827bfb26 (127.5.105.129:39737): RUNNING
329018d9fc43482486d43ea2cacc7bec (127.5.105.130:36671): not running
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap starting.
All reported replicas are:
A = 2d453524404b4643a1a2b6df91f0d19a
B = df5fcb65a69c43ebbeea4b62827bfb26
C = 329018d9fc43482486d43ea2cacc7bec
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 5 | Yes
B | A* B C | 1 | 5 | Yes
C | A B C | 1 | 5 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 22:54:35.756382 5542 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 22:54:35.894809 9833 raft_consensus.cc:1203] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260809 22:54:36.320812 9906 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260809 22:54:36.327749 9822 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:36.328390 9738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
7c95ed659ad44368ad3bcb9438fafad0 | 127.5.105.190:45807 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2d453524404b4643a1a2b6df91f0d19a | 127.5.105.131:33235 | HEALTHY | <none> | 1 | 0
329018d9fc43482486d43ea2cacc7bec | 127.5.105.130:36671 | HEALTHY | <none> | 0 | 0
df5fcb65a69c43ebbeea4b62827bfb26 | 127.5.105.129:39737 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260809 22:54:36.520983 5542 log_verifier.cc:126] Checking tablet 3b41d5fe85cf4ee3abf3f871f42c687b
I20260809 22:54:36.581687 5542 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 3b41d5fe85cf4ee3abf3f871f42c687b
I20260809 22:54:36.585582 5542 tablet_server.cc:178] TabletServer@127.5.105.129:0 shutting down...
I20260809 22:54:36.620224 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:36.620926 5542 tablet_replica.cc:330] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26: stopping tablet replica
I20260809 22:54:36.621512 5542 raft_consensus.cc:2229] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:36.622049 5542 raft_consensus.cc:2258] T 3b41d5fe85cf4ee3abf3f871f42c687b P df5fcb65a69c43ebbeea4b62827bfb26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:36.643376 5542 tablet_server.cc:194] TabletServer@127.5.105.129:0 shutdown complete.
I20260809 22:54:36.653869 5542 tablet_server.cc:178] TabletServer@127.5.105.130:0 shutting down...
I20260809 22:54:36.679535 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:36.680130 5542 tablet_replica.cc:330] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec: stopping tablet replica
I20260809 22:54:36.680665 5542 raft_consensus.cc:2229] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:36.681062 5542 raft_consensus.cc:2258] T 3b41d5fe85cf4ee3abf3f871f42c687b P 329018d9fc43482486d43ea2cacc7bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:36.701097 5542 tablet_server.cc:194] TabletServer@127.5.105.130:0 shutdown complete.
I20260809 22:54:36.711488 5542 tablet_server.cc:178] TabletServer@127.5.105.131:0 shutting down...
W20260809 22:54:36.722529 9874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.105.129:39737: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260809 22:54:36.728688 9874 consensus_peers.cc:487] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a -> Peer df5fcb65a69c43ebbeea4b62827bfb26 (127.5.105.129:39737): Couldn't send request to peer df5fcb65a69c43ebbeea4b62827bfb26. Status: Network error: Client connection negotiation failed: client connection to 127.5.105.129:39737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 22:54:36.738359 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:36.739032 5542 tablet_replica.cc:330] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a: stopping tablet replica
I20260809 22:54:36.739488 5542 raft_consensus.cc:2229] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:36.740211 5542 raft_consensus.cc:2258] T 3b41d5fe85cf4ee3abf3f871f42c687b P 2d453524404b4643a1a2b6df91f0d19a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:36.760800 5542 tablet_server.cc:194] TabletServer@127.5.105.131:0 shutdown complete.
I20260809 22:54:36.771885 5542 master.cc:537] Master@127.5.105.190:45807 shutting down...
I20260809 22:54:36.793504 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:36.793989 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:36.794320 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7c95ed659ad44368ad3bcb9438fafad0: stopping tablet replica
I20260809 22:54:36.812497 5542 master.cc:552] Master@127.5.105.190:45807 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (4870 ms)
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/1
I20260809 22:54:36.844540 5542 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.5.105.190:38571
I20260809 22:54:36.845623 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:36.851130 10196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:36.852167 10197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:36.854676 10199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:36.856125 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:36.856927 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:36.857091 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:36.857194 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316076857186 us; error 0 us; skew 500 ppm
I20260809 22:54:36.857668 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:36.860140 5542 webserver.cc:460] Webserver started at http://127.5.105.190:43065/ using document root <none> and password file <none>
I20260809 22:54:36.860543 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:36.860682 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:36.860883 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:36.861955 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/master-0-root/instance:
uuid: "b6209cf0dfbe449b9620806b4760940c"
format_stamp: "Formatted at 2026-08-09 22:54:36 on dist-test-slave-gp6n"
I20260809 22:54:36.866387 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 22:54:36.870213 10205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:36.871140 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260809 22:54:36.871491 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/master-0-root
uuid: "b6209cf0dfbe449b9620806b4760940c"
format_stamp: "Formatted at 2026-08-09 22:54:36 on dist-test-slave-gp6n"
I20260809 22:54:36.871868 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:36.891929 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:36.892992 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:36.931304 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.190:38571
I20260809 22:54:36.931429 10264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.190:38571 every 8 connection(s)
I20260809 22:54:36.935122 10265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:36.946254 10265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c: Bootstrap starting.
I20260809 22:54:36.950794 10265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:36.955040 10265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c: No bootstrap required, opened a new log
I20260809 22:54:36.957007 10265 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6209cf0dfbe449b9620806b4760940c" member_type: VOTER }
I20260809 22:54:36.957379 10265 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:36.957653 10265 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6209cf0dfbe449b9620806b4760940c, State: Initialized, Role: FOLLOWER
I20260809 22:54:36.958272 10265 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6209cf0dfbe449b9620806b4760940c" member_type: VOTER }
I20260809 22:54:36.958750 10265 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:54:36.958950 10265 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:54:36.959246 10265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:36.963706 10265 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6209cf0dfbe449b9620806b4760940c" member_type: VOTER }
I20260809 22:54:36.964254 10265 leader_election.cc:302] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6209cf0dfbe449b9620806b4760940c; no voters:
I20260809 22:54:36.965420 10265 leader_election.cc:288] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:54:36.965895 10268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:36.967279 10268 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [term 1 LEADER]: Becoming Leader. State: Replica: b6209cf0dfbe449b9620806b4760940c, State: Running, Role: LEADER
I20260809 22:54:36.967864 10268 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6209cf0dfbe449b9620806b4760940c" member_type: VOTER }
I20260809 22:54:36.968438 10265 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:54:36.972285 10269 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6209cf0dfbe449b9620806b4760940c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6209cf0dfbe449b9620806b4760940c" member_type: VOTER } }
I20260809 22:54:36.973839 10269 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:36.973121 10270 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6209cf0dfbe449b9620806b4760940c. Latest consensus state: current_term: 1 leader_uuid: "b6209cf0dfbe449b9620806b4760940c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6209cf0dfbe449b9620806b4760940c" member_type: VOTER } }
I20260809 22:54:36.975116 10270 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [sys.catalog]: This master's current role is: LEADER
I20260809 22:54:36.976785 10278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:54:36.981798 10278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:54:36.983534 5542 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 22:54:36.991175 10278 catalog_manager.cc:1347] Generated new cluster ID: c1984349fb074383b7f55ac23421ebac
I20260809 22:54:36.991459 10278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:54:37.014820 10278 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:54:37.016695 10278 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:54:37.042084 10278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c: Generated new TSK 0
I20260809 22:54:37.043040 10278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:54:37.049759 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:37.055958 10290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:37.056511 10291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:37.058394 10293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:37.059366 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:37.060303 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:37.060499 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:37.060669 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316077060642 us; error 0 us; skew 500 ppm
I20260809 22:54:37.061182 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:37.063616 5542 webserver.cc:460] Webserver started at http://127.5.105.129:42867/ using document root <none> and password file <none>
I20260809 22:54:37.064060 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:37.064232 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:37.064497 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:37.065505 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-0-root/instance:
uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd"
format_stamp: "Formatted at 2026-08-09 22:54:37 on dist-test-slave-gp6n"
I20260809 22:54:37.069876 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 22:54:37.073212 10299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:37.074012 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 22:54:37.074304 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-0-root
uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd"
format_stamp: "Formatted at 2026-08-09 22:54:37 on dist-test-slave-gp6n"
I20260809 22:54:37.074661 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:37.094256 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:37.095521 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:37.097020 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:54:37.099313 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:37.099534 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:37.099781 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:37.099942 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:37.141546 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.129:38733
I20260809 22:54:37.141664 10369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.129:38733 every 8 connection(s)
I20260809 22:54:37.146528 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:37.153867 10377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:37.154526 10378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:37.157405 10380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:37.158545 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:37.159525 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:37.159749 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:37.159924 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316077159911 us; error 0 us; skew 500 ppm
I20260809 22:54:37.160558 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:37.163576 10370 heartbeater.cc:347] Connected to a master server at 127.5.105.190:38571
I20260809 22:54:37.163641 5542 webserver.cc:460] Webserver started at http://127.5.105.130:42811/ using document root <none> and password file <none>
I20260809 22:54:37.164037 10370 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:37.164271 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:37.164551 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:37.164891 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:37.164897 10370 heartbeater.cc:511] Master 127.5.105.190:38571 requested a full tablet report, sending...
I20260809 22:54:37.166148 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-1-root/instance:
uuid: "35fc2cf2355145c9b2594d5eb01ba978"
format_stamp: "Formatted at 2026-08-09 22:54:37 on dist-test-slave-gp6n"
I20260809 22:54:37.167152 10226 ts_manager.cc:194] Registered new tserver with Master: 9b1ab6fc6e54438a9f6e9e9390ef09dd (127.5.105.129:38733)
I20260809 22:54:37.169322 10226 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37500
I20260809 22:54:37.171032 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:54:37.174652 10385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:37.175678 5542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 22:54:37.175982 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-1-root
uuid: "35fc2cf2355145c9b2594d5eb01ba978"
format_stamp: "Formatted at 2026-08-09 22:54:37 on dist-test-slave-gp6n"
I20260809 22:54:37.176262 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:37.206853 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:37.208109 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:37.209652 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:54:37.211949 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:37.212174 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:37.212419 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:37.212585 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:37.253204 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.130:42307
I20260809 22:54:37.253316 10453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.130:42307 every 8 connection(s)
I20260809 22:54:37.258035 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:37.264726 10457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:37.265777 10458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:37.268930 10462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:37.270032 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:37.271147 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:37.271404 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:37.271601 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316077271584 us; error 0 us; skew 500 ppm
I20260809 22:54:37.272267 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:37.272326 10454 heartbeater.cc:347] Connected to a master server at 127.5.105.190:38571
I20260809 22:54:37.272781 10454 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:37.273694 10454 heartbeater.cc:511] Master 127.5.105.190:38571 requested a full tablet report, sending...
I20260809 22:54:37.275295 5542 webserver.cc:460] Webserver started at http://127.5.105.131:43771/ using document root <none> and password file <none>
I20260809 22:54:37.275904 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:37.276101 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:37.275974 10226 ts_manager.cc:194] Registered new tserver with Master: 35fc2cf2355145c9b2594d5eb01ba978 (127.5.105.130:42307)
I20260809 22:54:37.276536 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:37.278020 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-2-root/instance:
uuid: "ef9bf565a83f4d9aae27f6544dea8d90"
format_stamp: "Formatted at 2026-08-09 22:54:37 on dist-test-slave-gp6n"
I20260809 22:54:37.278113 10226 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37506
I20260809 22:54:37.283267 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260809 22:54:37.286563 10467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:37.287292 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 22:54:37.287570 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-2-root
uuid: "ef9bf565a83f4d9aae27f6544dea8d90"
format_stamp: "Formatted at 2026-08-09 22:54:37 on dist-test-slave-gp6n"
I20260809 22:54:37.287822 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:37.307312 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:37.308436 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:37.309964 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:54:37.312206 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:37.312373 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:37.312642 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:37.312788 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:37.353374 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.131:42911
I20260809 22:54:37.353492 10543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.131:42911 every 8 connection(s)
I20260809 22:54:37.358234 5542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:54:37.365152 10547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:37.365692 10549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:54:37.369022 10552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:54:37.369670 5542 server_base.cc:1034] running on GCE node
I20260809 22:54:37.370841 5542 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 22:54:37.371107 5542 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 22:54:37.371357 5542 hybrid_clock.cc:648] HybridClock initialized: now 1786316077371332 us; error 0 us; skew 500 ppm
I20260809 22:54:37.372190 5542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:54:37.373827 10544 heartbeater.cc:347] Connected to a master server at 127.5.105.190:38571
I20260809 22:54:37.374217 10544 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:37.375166 10544 heartbeater.cc:511] Master 127.5.105.190:38571 requested a full tablet report, sending...
I20260809 22:54:37.375304 5542 webserver.cc:460] Webserver started at http://127.5.105.132:38259/ using document root <none> and password file <none>
I20260809 22:54:37.375962 5542 fs_manager.cc:359] Metadata directory not provided
I20260809 22:54:37.376194 5542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:54:37.376451 5542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:54:37.377627 10226 ts_manager.cc:194] Registered new tserver with Master: ef9bf565a83f4d9aae27f6544dea8d90 (127.5.105.131:42911)
I20260809 22:54:37.377840 5542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-3-root/instance:
uuid: "83237e2e733a4b83a150082561618a0d"
format_stamp: "Formatted at 2026-08-09 22:54:37 on dist-test-slave-gp6n"
I20260809 22:54:37.379948 10226 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37508
I20260809 22:54:37.383548 5542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 22:54:37.386833 10557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:37.387609 5542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 22:54:37.387907 5542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-3-root
uuid: "83237e2e733a4b83a150082561618a0d"
format_stamp: "Formatted at 2026-08-09 22:54:37 on dist-test-slave-gp6n"
I20260809 22:54:37.388156 5542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:54:37.409421 5542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:54:37.410663 5542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:54:37.412143 5542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:54:37.414315 5542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:54:37.414568 5542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:37.414814 5542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:54:37.414968 5542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:54:37.457085 5542 rpc_server.cc:307] RPC server started. Bound to: 127.5.105.132:40745
I20260809 22:54:37.457193 10634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.105.132:40745 every 8 connection(s)
I20260809 22:54:37.472581 10635 heartbeater.cc:347] Connected to a master server at 127.5.105.190:38571
I20260809 22:54:37.473006 10635 heartbeater.cc:464] Registering TS with master...
I20260809 22:54:37.473685 10635 heartbeater.cc:511] Master 127.5.105.190:38571 requested a full tablet report, sending...
I20260809 22:54:37.475595 10226 ts_manager.cc:194] Registered new tserver with Master: 83237e2e733a4b83a150082561618a0d (127.5.105.132:40745)
I20260809 22:54:37.476454 5542 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.016192421s
I20260809 22:54:37.477679 10226 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37522
I20260809 22:54:37.500597 5542 test_util.cc:268] Using random seed: 524388509
I20260809 22:54:37.522641 10226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37536:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 22:54:37.566861 10591 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9fdce442154bcd9799555a65ebac6f (DEFAULT_TABLE table=test-workload [id=552a9a54afb24142ba969afac64119b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:54:37.568367 10591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9fdce442154bcd9799555a65ebac6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:37.569918 10500 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9fdce442154bcd9799555a65ebac6f (DEFAULT_TABLE table=test-workload [id=552a9a54afb24142ba969afac64119b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:54:37.571244 10500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9fdce442154bcd9799555a65ebac6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:37.575362 10329 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9fdce442154bcd9799555a65ebac6f (DEFAULT_TABLE table=test-workload [id=552a9a54afb24142ba969afac64119b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:54:37.576577 10329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9fdce442154bcd9799555a65ebac6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:37.587536 10657 tablet_bootstrap.cc:492] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d: Bootstrap starting.
I20260809 22:54:37.589705 10658 tablet_bootstrap.cc:492] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90: Bootstrap starting.
I20260809 22:54:37.595017 10658 tablet_bootstrap.cc:654] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:37.595909 10657 tablet_bootstrap.cc:654] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:37.598902 10659 tablet_bootstrap.cc:492] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd: Bootstrap starting.
I20260809 22:54:37.600283 10658 tablet_bootstrap.cc:492] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90: No bootstrap required, opened a new log
I20260809 22:54:37.600849 10658 ts_tablet_manager.cc:1397] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260809 22:54:37.603017 10657 tablet_bootstrap.cc:492] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d: No bootstrap required, opened a new log
I20260809 22:54:37.603543 10657 ts_tablet_manager.cc:1397] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.013s
I20260809 22:54:37.603710 10658 raft_consensus.cc:348] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } }
I20260809 22:54:37.604511 10658 raft_consensus.cc:374] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:37.604741 10659 tablet_bootstrap.cc:654] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd: Neither blocks nor log segments found. Creating new log.
I20260809 22:54:37.604797 10658 raft_consensus.cc:729] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9bf565a83f4d9aae27f6544dea8d90, State: Initialized, Role: FOLLOWER
I20260809 22:54:37.605701 10658 consensus_queue.cc:260] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } }
I20260809 22:54:37.606846 10657 raft_consensus.cc:348] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } }
I20260809 22:54:37.607818 10657 raft_consensus.cc:374] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:37.608394 10544 heartbeater.cc:503] Master 127.5.105.190:38571 was elected leader, sending a full tablet report...
I20260809 22:54:37.608734 10658 ts_tablet_manager.cc:1428] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260809 22:54:37.609889 10657 raft_consensus.cc:729] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83237e2e733a4b83a150082561618a0d, State: Initialized, Role: FOLLOWER
I20260809 22:54:37.610571 10657 consensus_queue.cc:260] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } }
I20260809 22:54:37.613143 10635 heartbeater.cc:503] Master 127.5.105.190:38571 was elected leader, sending a full tablet report...
I20260809 22:54:37.613459 10659 tablet_bootstrap.cc:492] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd: No bootstrap required, opened a new log
I20260809 22:54:37.613585 10657 ts_tablet_manager.cc:1428] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260809 22:54:37.613919 10659 ts_tablet_manager.cc:1397] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260809 22:54:37.616761 10659 raft_consensus.cc:348] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } }
I20260809 22:54:37.617471 10659 raft_consensus.cc:374] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:54:37.617712 10659 raft_consensus.cc:729] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b1ab6fc6e54438a9f6e9e9390ef09dd, State: Initialized, Role: FOLLOWER
I20260809 22:54:37.618290 10659 consensus_queue.cc:260] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } }
I20260809 22:54:37.620570 10370 heartbeater.cc:503] Master 127.5.105.190:38571 was elected leader, sending a full tablet report...
I20260809 22:54:37.621495 10659 ts_tablet_manager.cc:1428] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260809 22:54:37.677132 10664 raft_consensus.cc:482] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:54:37.677707 10664 raft_consensus.cc:504] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } }
I20260809 22:54:37.679795 10664 leader_election.cc:288] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef9bf565a83f4d9aae27f6544dea8d90 (127.5.105.131:42911), 9b1ab6fc6e54438a9f6e9e9390ef09dd (127.5.105.129:38733)
I20260809 22:54:37.691756 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9fdce442154bcd9799555a65ebac6f" candidate_uuid: "83237e2e733a4b83a150082561618a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" is_pre_election: true
I20260809 22:54:37.692677 10512 raft_consensus.cc:2454] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83237e2e733a4b83a150082561618a0d in term 0.
I20260809 22:54:37.692903 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9fdce442154bcd9799555a65ebac6f" candidate_uuid: "83237e2e733a4b83a150082561618a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" is_pre_election: true
I20260809 22:54:37.693588 10340 raft_consensus.cc:2454] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83237e2e733a4b83a150082561618a0d in term 0.
I20260809 22:54:37.694522 10558 leader_election.cc:302] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83237e2e733a4b83a150082561618a0d, 9b1ab6fc6e54438a9f6e9e9390ef09dd; no voters:
I20260809 22:54:37.695230 10664 raft_consensus.cc:2781] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:54:37.695621 10664 raft_consensus.cc:482] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:54:37.695878 10664 raft_consensus.cc:3037] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:37.700516 10664 raft_consensus.cc:504] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } }
I20260809 22:54:37.702174 10664 leader_election.cc:288] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [CANDIDATE]: Term 1 election: Requested vote from peers ef9bf565a83f4d9aae27f6544dea8d90 (127.5.105.131:42911), 9b1ab6fc6e54438a9f6e9e9390ef09dd (127.5.105.129:38733)
I20260809 22:54:37.702811 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9fdce442154bcd9799555a65ebac6f" candidate_uuid: "83237e2e733a4b83a150082561618a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef9bf565a83f4d9aae27f6544dea8d90"
I20260809 22:54:37.703274 10512 raft_consensus.cc:3037] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:37.703205 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9fdce442154bcd9799555a65ebac6f" candidate_uuid: "83237e2e733a4b83a150082561618a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd"
I20260809 22:54:37.703825 10340 raft_consensus.cc:3037] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:54:37.708181 10512 raft_consensus.cc:2454] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83237e2e733a4b83a150082561618a0d in term 1.
I20260809 22:54:37.708398 10340 raft_consensus.cc:2454] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83237e2e733a4b83a150082561618a0d in term 1.
I20260809 22:54:37.709149 10562 leader_election.cc:302] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83237e2e733a4b83a150082561618a0d, ef9bf565a83f4d9aae27f6544dea8d90; no voters:
I20260809 22:54:37.709863 10664 raft_consensus.cc:2781] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:54:37.710891 10664 raft_consensus.cc:686] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [term 1 LEADER]: Becoming Leader. State: Replica: 83237e2e733a4b83a150082561618a0d, State: Running, Role: LEADER
I20260809 22:54:37.711704 10664 consensus_queue.cc:237] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } }
I20260809 22:54:37.717917 10226 catalog_manager.cc:5495] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83237e2e733a4b83a150082561618a0d (127.5.105.132). New cstate: current_term: 1 leader_uuid: "83237e2e733a4b83a150082561618a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } health_report { overall_health: UNKNOWN } } }
I20260809 22:54:37.963340 10591 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260809 22:54:37.963562 10415 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:37.974545 10329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:37.979192 10500 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:38.113116 10664 consensus_queue.cc:1035] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:54:38.115646 10664 consensus_queue.cc:1035] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
b6209cf0dfbe449b9620806b4760940c | 127.5.105.190:38571 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
35fc2cf2355145c9b2594d5eb01ba978 | 127.5.105.130:42307 | HEALTHY | <none> | 0 | 0
83237e2e733a4b83a150082561618a0d | 127.5.105.132:40745 | HEALTHY | <none> | 1 | 0
9b1ab6fc6e54438a9f6e9e9390ef09dd | 127.5.105.129:38733 | HEALTHY | <none> | 0 | 0
ef9bf565a83f4d9aae27f6544dea8d90 | 127.5.105.131:42911 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 4d9fdce442154bcd9799555a65ebac6f of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ef9bf565a83f4d9aae27f6544dea8d90 (127.5.105.131:42911): RUNNING
83237e2e733a4b83a150082561618a0d (127.5.105.132:40745): RUNNING [LEADER]
9b1ab6fc6e54438a9f6e9e9390ef09dd (127.5.105.129:38733): RUNNING
All reported replicas are:
A = ef9bf565a83f4d9aae27f6544dea8d90
B = 83237e2e733a4b83a150082561618a0d
C = 9b1ab6fc6e54438a9f6e9e9390ef09dd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 35fc2cf2355145c9b2594d5eb01ba978 | 127.5.105.130:42307 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 22:54:38.265535 5542 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 22:54:38.281458 10454 heartbeater.cc:503] Master 127.5.105.190:38571 was elected leader, sending a full tablet report...
I20260809 22:54:38.657832 10415 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:38.662111 10591 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260809 22:54:38.670238 10329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:38.676013 10500 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
b6209cf0dfbe449b9620806b4760940c | 127.5.105.190:38571 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
35fc2cf2355145c9b2594d5eb01ba978 | 127.5.105.130:42307 | HEALTHY | <none> | 0 | 0
83237e2e733a4b83a150082561618a0d | 127.5.105.132:40745 | HEALTHY | <none> | 1 | 0
9b1ab6fc6e54438a9f6e9e9390ef09dd | 127.5.105.129:38733 | HEALTHY | <none> | 0 | 0
ef9bf565a83f4d9aae27f6544dea8d90 | 127.5.105.131:42911 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 35fc2cf2355145c9b2594d5eb01ba978 | 127.5.105.130:42307 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260809 22:54:38.901225 5542 log_verifier.cc:126] Checking tablet 4d9fdce442154bcd9799555a65ebac6f
I20260809 22:54:38.962323 5542 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 4d9fdce442154bcd9799555a65ebac6f
I20260809 22:54:38.962925 5542 tablet_replica.cc:330] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d: stopping tablet replica
I20260809 22:54:38.963495 5542 raft_consensus.cc:2229] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:38.964339 5542 raft_consensus.cc:2258] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:39.204370 10500 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:39.205330 10591 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:39.213537 10415 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:39.214314 10329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
b6209cf0dfbe449b9620806b4760940c | 127.5.105.190:38571 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
35fc2cf2355145c9b2594d5eb01ba978 | 127.5.105.130:42307 | HEALTHY | <none> | 0 | 0
83237e2e733a4b83a150082561618a0d | 127.5.105.132:40745 | HEALTHY | <none> | 0 | 0
9b1ab6fc6e54438a9f6e9e9390ef09dd | 127.5.105.129:38733 | HEALTHY | <none> | 0 | 0
ef9bf565a83f4d9aae27f6544dea8d90 | 127.5.105.131:42911 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 4d9fdce442154bcd9799555a65ebac6f of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
ef9bf565a83f4d9aae27f6544dea8d90 (127.5.105.131:42911): RUNNING
83237e2e733a4b83a150082561618a0d (127.5.105.132:40745): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
9b1ab6fc6e54438a9f6e9e9390ef09dd (127.5.105.129:38733): RUNNING
All reported replicas are:
A = ef9bf565a83f4d9aae27f6544dea8d90
B = 83237e2e733a4b83a150082561618a0d
C = 9b1ab6fc6e54438a9f6e9e9390ef09dd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 35fc2cf2355145c9b2594d5eb01ba978 | 127.5.105.130:42307 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 22:54:39.433235 5542 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 22:54:39.806149 10415 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:39.829735 10591 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:39.839653 10329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:39.853874 10500 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
b6209cf0dfbe449b9620806b4760940c | 127.5.105.190:38571 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
35fc2cf2355145c9b2594d5eb01ba978 | 127.5.105.130:42307 | HEALTHY | <none> | 0 | 0
83237e2e733a4b83a150082561618a0d | 127.5.105.132:40745 | HEALTHY | <none> | 0 | 0
9b1ab6fc6e54438a9f6e9e9390ef09dd | 127.5.105.129:38733 | HEALTHY | <none> | 0 | 0
ef9bf565a83f4d9aae27f6544dea8d90 | 127.5.105.131:42911 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 4d9fdce442154bcd9799555a65ebac6f of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
ef9bf565a83f4d9aae27f6544dea8d90 (127.5.105.131:42911): RUNNING
83237e2e733a4b83a150082561618a0d (127.5.105.132:40745): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
9b1ab6fc6e54438a9f6e9e9390ef09dd (127.5.105.129:38733): RUNNING
All reported replicas are:
A = ef9bf565a83f4d9aae27f6544dea8d90
B = 83237e2e733a4b83a150082561618a0d
C = 9b1ab6fc6e54438a9f6e9e9390ef09dd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 35fc2cf2355145c9b2594d5eb01ba978 | 127.5.105.130:42307 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 22:54:40.057935 5542 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 22:54:40.061982 10811 raft_consensus.cc:482] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83237e2e733a4b83a150082561618a0d)
I20260809 22:54:40.062481 10811 raft_consensus.cc:504] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } }
I20260809 22:54:40.064342 10811 leader_election.cc:288] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef9bf565a83f4d9aae27f6544dea8d90 (127.5.105.131:42911), 83237e2e733a4b83a150082561618a0d (127.5.105.132:40745)
I20260809 22:54:40.083202 10604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9fdce442154bcd9799555a65ebac6f" candidate_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83237e2e733a4b83a150082561618a0d" is_pre_election: true
W20260809 22:54:40.084515 10300 leader_election.cc:341] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83237e2e733a4b83a150082561618a0d (127.5.105.132:40745): Illegal state: cannot vote while shut down
I20260809 22:54:40.087738 10814 raft_consensus.cc:482] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83237e2e733a4b83a150082561618a0d)
I20260809 22:54:40.088063 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9fdce442154bcd9799555a65ebac6f" candidate_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" is_pre_election: true
I20260809 22:54:40.088290 10814 raft_consensus.cc:504] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } }
I20260809 22:54:40.089341 10512 raft_consensus.cc:2454] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b1ab6fc6e54438a9f6e9e9390ef09dd in term 1.
I20260809 22:54:40.090611 10304 leader_election.cc:302] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b1ab6fc6e54438a9f6e9e9390ef09dd, ef9bf565a83f4d9aae27f6544dea8d90; no voters: 83237e2e733a4b83a150082561618a0d
I20260809 22:54:40.090844 10814 leader_election.cc:288] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83237e2e733a4b83a150082561618a0d (127.5.105.132:40745), 9b1ab6fc6e54438a9f6e9e9390ef09dd (127.5.105.129:38733)
I20260809 22:54:40.091488 10811 raft_consensus.cc:2781] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 22:54:40.091849 10811 raft_consensus.cc:482] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83237e2e733a4b83a150082561618a0d)
I20260809 22:54:40.092168 10811 raft_consensus.cc:3037] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:54:40.097817 10811 raft_consensus.cc:504] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } }
I20260809 22:54:40.099879 10811 leader_election.cc:288] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [CANDIDATE]: Term 2 election: Requested vote from peers ef9bf565a83f4d9aae27f6544dea8d90 (127.5.105.131:42911), 83237e2e733a4b83a150082561618a0d (127.5.105.132:40745)
I20260809 22:54:40.101001 10604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9fdce442154bcd9799555a65ebac6f" candidate_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83237e2e733a4b83a150082561618a0d"
I20260809 22:54:40.101001 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9fdce442154bcd9799555a65ebac6f" candidate_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ef9bf565a83f4d9aae27f6544dea8d90"
I20260809 22:54:40.101894 10512 raft_consensus.cc:3037] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 1 FOLLOWER]: Advancing to term 2
W20260809 22:54:40.102674 10300 leader_election.cc:341] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 83237e2e733a4b83a150082561618a0d (127.5.105.132:40745): Illegal state: cannot vote while shut down
I20260809 22:54:40.108741 10512 raft_consensus.cc:2454] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b1ab6fc6e54438a9f6e9e9390ef09dd in term 2.
I20260809 22:54:40.108965 10604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9fdce442154bcd9799555a65ebac6f" candidate_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83237e2e733a4b83a150082561618a0d" is_pre_election: true
W20260809 22:54:40.110991 10468 leader_election.cc:341] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83237e2e733a4b83a150082561618a0d (127.5.105.132:40745): Illegal state: cannot vote while shut down
I20260809 22:54:40.110913 10304 leader_election.cc:302] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b1ab6fc6e54438a9f6e9e9390ef09dd, ef9bf565a83f4d9aae27f6544dea8d90; no voters: 83237e2e733a4b83a150082561618a0d
I20260809 22:54:40.111986 10811 raft_consensus.cc:2781] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 2 FOLLOWER]: Leader election won for term 2
I20260809 22:54:40.113351 10811 raft_consensus.cc:686] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 2 LEADER]: Becoming Leader. State: Replica: 9b1ab6fc6e54438a9f6e9e9390ef09dd, State: Running, Role: LEADER
I20260809 22:54:40.114284 10811 consensus_queue.cc:237] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } }
I20260809 22:54:40.114692 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9fdce442154bcd9799555a65ebac6f" candidate_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" is_pre_election: true
I20260809 22:54:40.116096 10468 leader_election.cc:302] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef9bf565a83f4d9aae27f6544dea8d90; no voters: 83237e2e733a4b83a150082561618a0d, 9b1ab6fc6e54438a9f6e9e9390ef09dd
I20260809 22:54:40.116968 10814 raft_consensus.cc:2726] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 22:54:40.121646 10223 catalog_manager.cc:5495] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd reported cstate change: term changed from 1 to 2, leader changed from 83237e2e733a4b83a150082561618a0d (127.5.105.132) to 9b1ab6fc6e54438a9f6e9e9390ef09dd (127.5.105.129). New cstate: current_term: 2 leader_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83237e2e733a4b83a150082561618a0d" member_type: VOTER last_known_addr { host: "127.5.105.132" port: 40745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } health_report { overall_health: HEALTHY } } }
I20260809 22:54:40.510757 10591 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:40.519443 10415 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:40.523317 10500 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:40.526847 10329 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260809 22:54:40.550611 10300 consensus_queue.cc:579] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [LEADER]: The tablet replica hosted on peer 83237e2e733a4b83a150082561618a0d has failed
W20260809 22:54:40.551177 10300 consensus_peers.cc:487] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd -> Peer 83237e2e733a4b83a150082561618a0d (127.5.105.132:40745): Couldn't send request to peer 83237e2e733a4b83a150082561618a0d. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260809 22:54:40.680889 10512 raft_consensus.cc:1261] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 2 FOLLOWER]: Refusing update from remote peer 9b1ab6fc6e54438a9f6e9e9390ef09dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 22:54:40.682917 10817 consensus_queue.cc:1035] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 22:54:40.731181 10340 consensus_queue.cc:237] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } }
I20260809 22:54:40.735826 10511 raft_consensus.cc:1261] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 2 FOLLOWER]: Refusing update from remote peer 9b1ab6fc6e54438a9f6e9e9390ef09dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260809 22:54:40.737224 10811 consensus_queue.cc:1035] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 22:54:40.743366 10817 raft_consensus.cc:2932] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 83237e2e733a4b83a150082561618a0d (127.5.105.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } } }
I20260809 22:54:40.744923 10511 raft_consensus.cc:2932] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 83237e2e733a4b83a150082561618a0d (127.5.105.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } } }
I20260809 22:54:40.752753 10206 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4d9fdce442154bcd9799555a65ebac6f with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 22:54:40.755885 10223 catalog_manager.cc:5495] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 reported cstate change: config changed from index -1 to 3, VOTER 83237e2e733a4b83a150082561618a0d (127.5.105.132) evicted. New cstate: current_term: 2 leader_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } } }
I20260809 22:54:40.767933 10591 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9fdce442154bcd9799555a65ebac6f with delete_type TABLET_DATA_TOMBSTONED (TS 83237e2e733a4b83a150082561618a0d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55570
I20260809 22:54:40.772459 10340 consensus_queue.cc:237] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } } peers { permanent_uuid: "35fc2cf2355145c9b2594d5eb01ba978" member_type: NON_VOTER last_known_addr { host: "127.5.105.130" port: 42307 } attrs { promote: true } }
I20260809 22:54:40.776885 10867 ts_tablet_manager.cc:1905] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 22:54:40.782020 10511 raft_consensus.cc:1261] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 2 FOLLOWER]: Refusing update from remote peer 9b1ab6fc6e54438a9f6e9e9390ef09dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
b6209cf0dfbe449b9620806b4760940c | 127.5.105.190:38571 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
I20260809 22:54:40.784178 10817 consensus_queue.cc:1035] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
35fc2cf2355145c9b2594d5eb01ba978 | 127.5.105.130:42307 | HEALTHY | <none> | 0 | 0
83237e2e733a4b83a150082561618a0d | 127.5.105.132:40745 | HEALTHY | <none> | 0 | 0
9b1ab6fc6e54438a9f6e9e9390ef09dd | 127.5.105.129:38733 | HEALTHY | <none> | 1 | 0
ef9bf565a83f4d9aae27f6544dea8d90 | 127.5.105.131:42911 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 4d9fdce442154bcd9799555a65ebac6f of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
ef9bf565a83f4d9aae27f6544dea8d90 (127.5.105.131:42911): RUNNING
83237e2e733a4b83a150082561618a0d (127.5.105.132:40745): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
9b1ab6fc6e54438a9f6e9e9390ef09dd (127.5.105.129:38733): RUNNING [LEADER]
All reported replicas are:
A = ef9bf565a83f4d9aae27f6544dea8d90
B = 83237e2e733a4b83a150082561618a0d
C = 9b1ab6fc6e54438a9f6e9e9390ef09dd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
I20260809 22:54:40.791376 10811 raft_consensus.cc:2932] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER 35fc2cf2355145c9b2594d5eb01ba978 (127.5.105.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } } peers { permanent_uuid: "35fc2cf2355145c9b2594d5eb01ba978" member_type: NON_VOTER last_known_addr { host: "127.5.105.130" port: 42307 } attrs { promote: true } } }
I20260809 22:54:40.792676 10867 ts_tablet_manager.cc:1918] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260809 22:54:40.793079 10867 log.cc:1196] T 4d9fdce442154bcd9799555a65ebac6f P 83237e2e733a4b83a150082561618a0d: Deleting WAL directory at /tmp/dist-test-tasktYWdVH/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786316013627029-5542-0/minicluster-data/ts-3-root/wals/4d9fdce442154bcd9799555a65ebac6f
I20260809 22:54:40.793275 10511 raft_consensus.cc:2932] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER 35fc2cf2355145c9b2594d5eb01ba978 (127.5.105.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } } peers { permanent_uuid: "35fc2cf2355145c9b2594d5eb01ba978" member_type: NON_VOTER last_known_addr { host: "127.5.105.130" port: 42307 } attrs { promote: true } } }
I20260809 22:54:40.794809 10206 catalog_manager.cc:4841] TS 83237e2e733a4b83a150082561618a0d (127.5.105.132:40745): tablet 4d9fdce442154bcd9799555a65ebac6f (table test-workload [id=552a9a54afb24142ba969afac64119b8]) successfully deleted
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
W20260809 22:54:40.796468 10303 consensus_peers.cc:487] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd -> Peer 35fc2cf2355145c9b2594d5eb01ba978 (127.5.105.130:42307): Couldn't send request to peer 35fc2cf2355145c9b2594d5eb01ba978. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d9fdce442154bcd9799555a65ebac6f. This is attempt 1: this message will repeat every 5th retry.
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 35fc2cf2355145c9b2594d5eb01ba978 | 127.5.105.130:42307 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 22:54:40.803826 5542 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 22:54:40.804142 10206 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4d9fdce442154bcd9799555a65ebac6f with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260809 22:54:40.808128 10222 catalog_manager.cc:5495] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 reported cstate change: config changed from index 3 to 4, NON_VOTER 35fc2cf2355145c9b2594d5eb01ba978 (127.5.105.130) added. New cstate: current_term: 2 leader_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } } peers { permanent_uuid: "35fc2cf2355145c9b2594d5eb01ba978" member_type: NON_VOTER last_known_addr { host: "127.5.105.130" port: 42307 } attrs { promote: true } } }
I20260809 22:54:41.275768 10879 ts_tablet_manager.cc:927] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978: Initiating tablet copy from peer 9b1ab6fc6e54438a9f6e9e9390ef09dd (127.5.105.129:38733)
I20260809 22:54:41.277661 10879 tablet_copy_client.cc:316] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978: tablet copy: Beginning tablet copy session from remote peer at address 127.5.105.129:38733
I20260809 22:54:41.295682 10351 tablet_copy_service.cc:140] P 9b1ab6fc6e54438a9f6e9e9390ef09dd: Received BeginTabletCopySession request for tablet 4d9fdce442154bcd9799555a65ebac6f from peer 35fc2cf2355145c9b2594d5eb01ba978 ({username='slave'} at 127.0.0.1:51660)
I20260809 22:54:41.296257 10351 tablet_copy_service.cc:161] P 9b1ab6fc6e54438a9f6e9e9390ef09dd: Beginning new tablet copy session on tablet 4d9fdce442154bcd9799555a65ebac6f from peer 35fc2cf2355145c9b2594d5eb01ba978 at {username='slave'} at 127.0.0.1:51660: session id = 35fc2cf2355145c9b2594d5eb01ba978-4d9fdce442154bcd9799555a65ebac6f
I20260809 22:54:41.303583 10351 tablet_copy_source_session.cc:205] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 22:54:41.306878 10879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9fdce442154bcd9799555a65ebac6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:54:41.320763 10879 tablet_copy_client.cc:791] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978: tablet copy: Starting download of 0 data blocks...
I20260809 22:54:41.321368 10879 tablet_copy_client.cc:655] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978: tablet copy: Starting download of 1 WAL segments...
I20260809 22:54:41.325827 10879 tablet_copy_client.cc:531] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 22:54:41.333640 10879 tablet_bootstrap.cc:492] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978: Bootstrap starting.
I20260809 22:54:41.354470 10879 tablet_bootstrap.cc:492] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 22:54:41.355544 10879 tablet_bootstrap.cc:492] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978: Bootstrap complete.
I20260809 22:54:41.356544 10879 ts_tablet_manager.cc:1397] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978: Time spent bootstrapping tablet: real 0.023s user 0.024s sys 0.001s
I20260809 22:54:41.360370 10879 raft_consensus.cc:348] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } } peers { permanent_uuid: "35fc2cf2355145c9b2594d5eb01ba978" member_type: NON_VOTER last_known_addr { host: "127.5.105.130" port: 42307 } attrs { promote: true } }
I20260809 22:54:41.361574 10879 raft_consensus.cc:729] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 35fc2cf2355145c9b2594d5eb01ba978, State: Initialized, Role: LEARNER
I20260809 22:54:41.362413 10879 consensus_queue.cc:260] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } } peers { permanent_uuid: "35fc2cf2355145c9b2594d5eb01ba978" member_type: NON_VOTER last_known_addr { host: "127.5.105.130" port: 42307 } attrs { promote: true } }
I20260809 22:54:41.370302 10879 ts_tablet_manager.cc:1428] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260809 22:54:41.372087 10351 tablet_copy_service.cc:342] P 9b1ab6fc6e54438a9f6e9e9390ef09dd: Request end of tablet copy session 35fc2cf2355145c9b2594d5eb01ba978-4d9fdce442154bcd9799555a65ebac6f received from {username='slave'} at 127.0.0.1:51660
I20260809 22:54:41.372645 10351 tablet_copy_service.cc:433] P 9b1ab6fc6e54438a9f6e9e9390ef09dd: ending tablet copy session 35fc2cf2355145c9b2594d5eb01ba978-4d9fdce442154bcd9799555a65ebac6f on tablet 4d9fdce442154bcd9799555a65ebac6f with peer 35fc2cf2355145c9b2594d5eb01ba978
I20260809 22:54:41.404472 10415 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:41.411433 10591 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:41.412151 10329 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260809 22:54:41.418051 10500 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
b6209cf0dfbe449b9620806b4760940c | 127.5.105.190:38571 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
35fc2cf2355145c9b2594d5eb01ba978 | 127.5.105.130:42307 | HEALTHY | <none> | 0 | 0
83237e2e733a4b83a150082561618a0d | 127.5.105.132:40745 | HEALTHY | <none> | 0 | 0
9b1ab6fc6e54438a9f6e9e9390ef09dd | 127.5.105.129:38733 | HEALTHY | <none> | 1 | 0
ef9bf565a83f4d9aae27f6544dea8d90 | 127.5.105.131:42911 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 4d9fdce442154bcd9799555a65ebac6f of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ef9bf565a83f4d9aae27f6544dea8d90 (127.5.105.131:42911): RUNNING
9b1ab6fc6e54438a9f6e9e9390ef09dd (127.5.105.129:38733): RUNNING [LEADER]
35fc2cf2355145c9b2594d5eb01ba978 (127.5.105.130:42307): RUNNING [NONVOTER]
All reported replicas are:
A = ef9bf565a83f4d9aae27f6544dea8d90
B = 9b1ab6fc6e54438a9f6e9e9390ef09dd
C = 35fc2cf2355145c9b2594d5eb01ba978
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* C~ | 2 | 4 | Yes
B | A B* C~ | 2 | 4 | Yes
C | A B C~ | 2 | 4 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 83237e2e733a4b83a150082561618a0d | 127.5.105.132:40745 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260809 22:54:41.652161 5542 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260809 22:54:41.686652 10425 raft_consensus.cc:1203] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.3->[2.4-2.4] Dedup: 2.4->[]
I20260809 22:54:42.303378 10910 raft_consensus.cc:1050] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd: attempting to promote NON_VOTER 35fc2cf2355145c9b2594d5eb01ba978 to VOTER
I20260809 22:54:42.305620 10910 consensus_queue.cc:237] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } } peers { permanent_uuid: "35fc2cf2355145c9b2594d5eb01ba978" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42307 } attrs { promote: false } }
I20260809 22:54:42.312175 10425 raft_consensus.cc:1261] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978 [term 2 LEARNER]: Refusing update from remote peer 9b1ab6fc6e54438a9f6e9e9390ef09dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 22:54:42.313551 10511 raft_consensus.cc:1261] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 2 FOLLOWER]: Refusing update from remote peer 9b1ab6fc6e54438a9f6e9e9390ef09dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 22:54:42.313570 10928 consensus_queue.cc:1035] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35fc2cf2355145c9b2594d5eb01ba978" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 22:54:42.315387 10910 consensus_queue.cc:1035] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 22:54:42.321683 10928 raft_consensus.cc:2932] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, 35fc2cf2355145c9b2594d5eb01ba978 (127.5.105.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } } peers { permanent_uuid: "35fc2cf2355145c9b2594d5eb01ba978" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42307 } attrs { promote: false } } }
I20260809 22:54:42.325702 10425 raft_consensus.cc:2932] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, 35fc2cf2355145c9b2594d5eb01ba978 (127.5.105.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } } peers { permanent_uuid: "35fc2cf2355145c9b2594d5eb01ba978" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42307 } attrs { promote: false } } }
I20260809 22:54:42.327227 10512 raft_consensus.cc:2932] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, 35fc2cf2355145c9b2594d5eb01ba978 (127.5.105.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } } peers { permanent_uuid: "35fc2cf2355145c9b2594d5eb01ba978" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42307 } attrs { promote: false } } }
I20260809 22:54:42.335672 10223 catalog_manager.cc:5495] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd reported cstate change: config changed from index 4 to 5, 35fc2cf2355145c9b2594d5eb01ba978 (127.5.105.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef9bf565a83f4d9aae27f6544dea8d90" member_type: VOTER last_known_addr { host: "127.5.105.131" port: 42911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b1ab6fc6e54438a9f6e9e9390ef09dd" member_type: VOTER last_known_addr { host: "127.5.105.129" port: 38733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35fc2cf2355145c9b2594d5eb01ba978" member_type: VOTER last_known_addr { host: "127.5.105.130" port: 42307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 22:54:42.443449 10415 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:42.454094 10591 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:42.457926 10500 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260809 22:54:42.461011 10329 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
b6209cf0dfbe449b9620806b4760940c | 127.5.105.190:38571 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
35fc2cf2355145c9b2594d5eb01ba978 | 127.5.105.130:42307 | HEALTHY | <none> | 0 | 0
83237e2e733a4b83a150082561618a0d | 127.5.105.132:40745 | HEALTHY | <none> | 0 | 0
9b1ab6fc6e54438a9f6e9e9390ef09dd | 127.5.105.129:38733 | HEALTHY | <none> | 1 | 0
ef9bf565a83f4d9aae27f6544dea8d90 | 127.5.105.131:42911 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 83237e2e733a4b83a150082561618a0d | 127.5.105.132:40745 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260809 22:54:42.668563 5542 log_verifier.cc:126] Checking tablet 4d9fdce442154bcd9799555a65ebac6f
I20260809 22:54:42.731746 5542 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 4d9fdce442154bcd9799555a65ebac6f
I20260809 22:54:42.733829 5542 tablet_server.cc:178] TabletServer@127.5.105.129:0 shutting down...
I20260809 22:54:42.757751 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:42.758378 5542 tablet_replica.cc:330] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd: stopping tablet replica
I20260809 22:54:42.759016 5542 raft_consensus.cc:2229] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 2 LEADER]: Raft consensus shutting down.
I20260809 22:54:42.759811 5542 raft_consensus.cc:2258] T 4d9fdce442154bcd9799555a65ebac6f P 9b1ab6fc6e54438a9f6e9e9390ef09dd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:42.772104 5542 tablet_server.cc:194] TabletServer@127.5.105.129:0 shutdown complete.
I20260809 22:54:42.782832 5542 tablet_server.cc:178] TabletServer@127.5.105.130:0 shutting down...
I20260809 22:54:42.804319 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:42.804983 5542 tablet_replica.cc:330] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978: stopping tablet replica
I20260809 22:54:42.805496 5542 raft_consensus.cc:2229] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:42.805970 5542 raft_consensus.cc:2258] T 4d9fdce442154bcd9799555a65ebac6f P 35fc2cf2355145c9b2594d5eb01ba978 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:42.826164 5542 tablet_server.cc:194] TabletServer@127.5.105.130:0 shutdown complete.
I20260809 22:54:42.835421 5542 tablet_server.cc:178] TabletServer@127.5.105.131:0 shutting down...
I20260809 22:54:42.855201 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:42.855728 5542 tablet_replica.cc:330] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90: stopping tablet replica
I20260809 22:54:42.856257 5542 raft_consensus.cc:2229] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 22:54:42.857071 5542 raft_consensus.cc:2258] T 4d9fdce442154bcd9799555a65ebac6f P ef9bf565a83f4d9aae27f6544dea8d90 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:42.877787 5542 tablet_server.cc:194] TabletServer@127.5.105.131:0 shutdown complete.
I20260809 22:54:42.887079 5542 tablet_server.cc:178] TabletServer@127.5.105.132:0 shutting down...
I20260809 22:54:42.906605 5542 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 22:54:42.923956 5542 tablet_server.cc:194] TabletServer@127.5.105.132:0 shutdown complete.
I20260809 22:54:42.933917 5542 master.cc:537] Master@127.5.105.190:38571 shutting down...
I20260809 22:54:42.951922 5542 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:54:42.952521 5542 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:54:42.953059 5542 tablet_replica.cc:330] T 00000000000000000000000000000000 P b6209cf0dfbe449b9620806b4760940c: stopping tablet replica
I20260809 22:54:42.972558 5542 master.cc:552] Master@127.5.105.190:38571 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (6154 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (11025 ms total)
[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (69206 ms total)
[ PASSED ] 11 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
[ FAILED ] 1 test, listed below:
[ FAILED ] TsTabletManagerITest.TestTableStats
1 FAILED TEST
I20260809 22:54:43.007851 5542 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~11 seconds ago
I20260809 22:54:43.008211 5542 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tserver/heartbeater.cc:644: suppressed but not reported on 100 messages since previous log ~60 seconds ago
I20260809 22:54:43.008528 5542 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tablet/tablet.cc:2369: suppressed but not reported on 9 messages since previous log ~15 seconds ago